对比全局与成对评分，发现各有优劣，选评估方式要看任务需求。
Confidence and Stability of Global and Pairwise Scores in NLP Evaluation
- 用全局分数和成对比较模型分析模型性能，验证两种方法的可靠性。
- 全局分数易低估有罕见大错的强模型，而成对比较更擅长发现弱分高质模型。
- 适合生成类任务或指标模糊场景，尤其推荐用于文本生成评估。
随着指令微调语言模型能力提升，自然语言处理的基准测试正从传统全局评分（如GLUE、BIG-bench、SWE-bench）转向成对比较排行榜（如LMSYS Arena）。本文通过在合成与真实数据集上的计算实验，研究了全局分数与成对比较各自的优缺点，以辅助选择合适的模型评估策略。使用标准全局指标及流行的Bradley-Terry模型进行成对比较，结果表明：全局分数虽能提供更可靠的总体排名，但会低估存在罕见重大错误或低置信度的强模型；而成对比较在识别全局得分较低但表现突出的模型方面更具优势，尤其适用于质量难以定义的任务（如文本生成），尽管当模型间胜负频繁时需更多比较才能收敛。代码与数据已开源。
原文摘要 · Abstract (English)
With the advent of highly capable instruction-tuned neural language models, benchmarking in natural language processing (NLP) is increasingly shifting towards pairwise comparison leaderboards, such as LMSYS Arena, from traditional global pointwise scores (e.g., GLUE, BIG-bench, SWE-bench). This paper empirically investigates the strengths and weaknesses of both global scores and pairwise comparisons to aid decision-making in selecting appropriate model evaluation strategies. Through computational experiments on synthetic and real-world datasets using standard global metrics and the popular Bradley-Terry model for pairwise comparisons, we found that while global scores provide more reliable overall rankings, they can underestimate strong models with rare, significant errors or low confidence. Conversely, pairwise comparisons are particularly effective for identifying strong contenders among models with lower global scores, especially where quality metrics are hard to define (e.g., text generation), though they require more comparisons to converge if ties are frequent. Our code and data are available at https://github.com/HSPyroblast/srw-ranking under a permissive license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。