用推理模型提升评估能力，多花算力让评估更准。
Scaling Evaluation-time Compute with Reasoning Models as Evaluators
- 用长思维链模型做评估，可逐步提升判断精度。
- 评估时多生成推理步骤，准确率随计算量线性上升。
- 适合想提升模型评估质量的研究者或开发者。
随着语言模型输出愈发自然，其质量评估变得日益困难。同时，通过增加测试阶段的计算量来延长模型‘思考’时间，已被证明能有效解决数学和代码等领域的难题。这引发了一个问题：能否通过增加测试阶段的计算量来提升语言模型的评估能力？为此，我们研究了使用具备长思维链推理能力的语言模型作为评估器的方法。具体而言，我们探索了两种利用更多测试阶段计算量的策略：（1）采用推理模型作为评估器；（2）引导这些模型不仅评估整体回答（结果评估），还逐步分析每个推理步骤（过程评估）。实验表明，评估器性能随推理文本长度增加而持续提升，与生成阶段的趋势一致。进一步地，我们使用更精准的评估器对多个生成结果进行重排序，发现增加评估阶段的计算量，在提升模型解题能力方面，效果可媲美增加生成阶段的计算量。
原文摘要 · Abstract (English)
As language model (LM) outputs get more and more natural, it is becoming more difficult than ever to evaluate their quality. Simultaneously, increasing LMs' "thinking" time through scaling test-time compute has proven an effective technique to solve challenging problems in domains such as math and code. This raises a natural question: can an LM's evaluation capability also be improved by spending more test-time compute? To answer this, we investigate employing reasoning models-LMs that natively generate long chain-of-thought reasoning-as evaluators. Specifically, we examine methods to leverage more test-time compute by (1) using reasoning models, and (2) prompting these models to evaluate not only the response as a whole (i.e., outcome evaluation) but also assess each step in the response separately (i.e., process evaluation). In experiments, we observe that the evaluator's performance improves monotonically when generating more reasoning tokens, similar to the trends observed in LM-based generation. Furthermore, we use these more accurate evaluators to rerank multiple generations, and demonstrate that spending more compute at evaluation time can be as effective as using more compute at generation time in improving an LM's problem-solving capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。