提出性能均匀性指标,让基准测试更可靠
The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
- 用分布视角衡量模型在子任务上的表现均匀性
- 19个基准测试中,低均匀性导致整体准确率失真
- 建议报告准确率时附带均匀性,适合评估者参考
基准测试塑造了对模型能力的科学判断,并引导模型发展,形成正向反馈循环:更强大的基准推动更好模型,而更好模型又需要更具区分性的基准。因此,确保基准可靠性对可信评估和有意义进展至关重要。本文从分布视角研究基准可靠性,提出‘基准和谐度’(benchmark harmony),用于衡量模型在基准各子域上性能分布的均匀性。高和谐度意味着模型在各子域上表现均衡,整体指标更能反映真实综合能力。我们在19个多项选择题基准和5个模型家族上,将每个基准映射到由平均和谐度与方差构成的平面中,高均值与低方差代表更可靠的评估。分析表明,不均匀的基准可能产生误导性结果——例如,ARC-Easy被生物学概念题主导,掩盖了地理、物理、化学与环境科学等关键子域的表现。我们建议在报告准确率的同时纳入和谐度,将评估从单一平均转向更具分布鲁棒性的性能测量。
原文摘要 · Abstract (English)
Benchmarks shape scientific conclusions about model capabilities and steer model development. This creates a feedback loop: stronger benchmarks drive better models, and better models demand more discriminative benchmarks. Ensuring benchmark reliability is therefore essential for trustworthy evaluation and meaningful progress. In this work, we study benchmark reliability from a distributional perspective and introduce benchmark harmony, which measures how uniformly a model's performance is distributed across the subdomains of a benchmark. We posit that high harmony is a desirable benchmark property, indicating that the aggregate metric reflects uniform competence across subdomains. Across 19 multiple-choice benchmarks and five model families, we map each benchmark onto a mean-variance plane of harmony computed across models, where high mean and low variance signal more reliable evaluation. Our analysis shows that less harmonious benchmarks can give misleading results, since overall accuracy may be disproportionately influenced by specific subdomains. For instance, ARC-Easy is overwhelmed by questions on Biological Concepts, overshadowing other critical subdomains such as Geography, Physics, Chemistry, and Environmental Science. By recommending that harmony should be reported alongside accuracy, we reframe evaluation from simple performance averages to a more robust, distributionally reliable measurement of performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。