测试大模型当裁判在推理时的评估效果，发现它适合排序答案但难改进生成过程。
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
- 用大模型自动生成评价和解释，代替传统奖励模型做评估
- 在重排答案任务中表现接近奖励模型，但在搜索优化中明显落后
- 生成的自然语言批评无法有效指导模型改进输出，适合研究自动评估者
测试时计算扩展（test-time scaling）通常依赖外部非生成式评估器（如奖励模型）。而训练用于生成自然语言评价与解释的大型语言模型（LLM-judges）在自动评估中日益流行。尽管判官模型在实践中取得成功，其在测试时扩展场景中的有效性仍不清楚。本文提出JETTS基准，评估10个不同规模的判官模型（7B-70B参数）在3个领域（数学推理、代码生成、指令遵循）下的表现，涵盖三种任务设置：响应重排、步骤级束搜索和基于批判的响应优化。实验覆盖8个基础生成器模型（6.7B-72B参数）。结果显示，判官在重排任务中可与结果型奖励模型竞争，但在束搜索过程中始终逊于过程型奖励模型。此外，虽然判官能生成自然语言批评，但这些批评目前无法有效引导生成器改善输出。
原文摘要 · Abstract (English)
Scaling test-time computation, or affording a generator large language model (LLM) extra compute during inference, typically employs the help of external non-generative evaluators (i.e., reward models). Concurrently, LLM-judges, models trained to generate evaluations and critiques (explanations) in natural language, are becoming increasingly popular in automatic evaluation. Despite judge empirical successes, their effectiveness as evaluators in test-time scaling settings is largely unknown. In this paper, we introduce the Judge Evaluation for Test-Time Scaling (JETTS) benchmark, which evaluates judge performance in three domains (math reasoning, code generation, and instruction following) under three task settings: response reranking, step-level beam search, and critique-based response refinement. We evaluate 10 different judge models (7B-70B parameters) for 8 different base generator models (6.7B-72B parameters). Our benchmark shows that while judges are competitive with outcome reward models in reranking, they are consistently worse than process reward models in beam search procedures. Furthermore, though unique to LLM-judges, their natural language critiques are currently ineffective in guiding the generator towards better responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。