发现大模型评判存在非传递性，影响排名可靠性。
Investigating Non-Transitivity in LLM-as-a-Judge
- 用循环赛+布拉德利-特里模型替代基准对比，提升评估稳定性
- 新方法使与Chatbot Arena的斯皮尔曼相关性提升至96.4%
- 提出Swim动态匹配机制，在效率与效果间取得平衡
基于大语言模型的自动评估已成为衡量大模型指令遵循能力的标准方法。当前主流的成对比较法依赖于偏好传递性的假设，但该假设尚未被充分验证。本研究在AlpacaEval框架下探究了大模型评判中的非传递性现象及其对模型排名的影响。结果表明，大模型裁判确实存在非传递偏好，导致排名结果高度依赖基准模型的选择。为此，我们提出采用循环赛结合布拉德利-特里偏好模型的方法，显著提升了排名可靠性：与Chatbot Arena的斯皮尔曼相关性从95.0%提升至96.4%，肯德尔相关性从82.1%提升至86.3%。为降低循环赛的计算开销，我们进一步设计了动态匹配的Swim锦标赛机制，既保留了循环赛的优势，又保证了计算效率。
原文摘要 · Abstract (English)
Automatic evaluation methods based on large language models (LLMs) are emerging as the standard tool for assessing the instruction-following abilities of LLM-based agents. The most common method in this paradigm, pairwise comparisons with a baseline model, critically depends on the assumption of transitive preferences. However, the validity of this assumption remains largely unexplored. In this study, we investigate the presence of non-transitivity within the AlpacaEval framework and analyze its effects on model rankings. We find that LLM judges exhibit non-transitive preferences, leading to rankings that are sensitive to the choice of the baseline model. To mitigate this issue, we show that round-robin tournaments combined with Bradley-Terry models of preference can produce more reliable rankings. Notably, our method increases both the Spearman correlation and the Kendall correlation with Chatbot Arena (95.0% -> 96.4% and 82.1% -> 86.3% respectively). To address the computational cost of round-robin tournaments, we propose Swiss-Wise Iterative Matchmaking (Swim) tournaments, using a dynamic matching strategy to capture the benefits of round-robin tournaments while maintaining computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。