LLM评分器看似可靠，实则难选最优回复。
When LLM Judge Scores Look Good but Best-of-N Decisions Fail
- 用局部对比判断替代全局评分，提升选择精度
- 全局相关性0.47，但实际选优仅达理想21%
- 配对打分可减少67%的平局，恢复61%选优能力
大型语言模型常被用作评分器，通过与参考标签的相关性来验证效果。但在实际的best-of-n选择任务中，这种全局指标可能具有误导性。在包含5,000个提示的Chatbot Arena best-of-4基准测试中，一个全局相关性为0.47的评分器，仅能捕捉到完美选择相比随机选择所带来改进的21.0%。这是因为全局一致性主要受提示级基线效应驱动，而选择依赖于提示内排序：提示内相关性仅为0.27，且点对点评分导致67%的成对比较出现平局。在匹配对的best-of-2审计中，显式成对评判显著恢复信号，使恢复率从21.1%提升至61.2%。因此，评估基于评分器的选择时，应报告提示内信号、平局率和恢复率/顶级准确率，而非仅看全局一致度。
原文摘要 · Abstract (English)
Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt. In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。