用淘汰赛方式直接对比模型，更高效可靠地评估大模型性能。
Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons
- 采用淘汰赛结构进行模型间直接对比，无需基准输出。
- 相同可靠性下比较次数减少，小数据集也能稳定排名。
- 适合需要快速筛选模型的研究与工业场景。
随着大语言模型在各领域扩展，模型评判系统变得至关重要。现有基准多通过对比系统输出与基线实现，但该方法可靠性较低。本文提出Arena-Lite，基于头对头直接比较构建淘汰赛结构，无需基线输出，显著减少比较次数，提升排名可靠性。通过两项实验：（1）受控随机建模，（2）真实大模型评判的实证验证，结果表明，即使在小数据集或弱评判者条件下，Arena-Lite仍能保持更高可靠性。项目提供可交互网页演示与开源代码，推动研究与产业界模型选型效率提升。演示与代码已发布于HuggingFace：https://huggingface.co/spaces/NCSOFT/ArenaLite。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) expand across domains, LLM judges have become essential for systems evaluation. Current benchmarks typically compare system outputs against baselines. This baseline-mediated approach, though convenient, yields lower reliability than direct comparison between systems. We propose Arena-Lite which integrates tournament structure on top of head-to-head comparison. The application of a tournament structure and direct comparison eliminates the need for baseline outputs, reduces the number of required comparisons, and allows higher reliability in system rankings. We conducted two experiments: (1) controlled stochastic modeling and (2) empirical validation with a real LLM judge. Those experiments collectively demonstrate that Arena-Lite consistently achieves higher reliability with fewer comparisons, even with smaller datasets or weaker judges. We release an easy-to-use web demonstration and code to foster adoption of Arena-Lite, streamlining model selection across research and industry communities. Arena-Lite demo and code are available on \href{https://huggingface.co/spaces/NCSOFT/ArenaLite}{https://huggingface.co/spaces/NCSOFT/ArenaLite}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。