arXiv:2608.21382cs.AI2026-08

大模型评测结果受评测方式影响极大,同一模型得分可差58个百分点。

There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items

论文配图:There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items
图 1 · 摘自论文原文
  • 通过26种评测配置对比12个模型在3679题上的表现,发现评测细节决定得分
  • 模型得分范围达31%至89%,关键差距来自对少数敏感题目处理不同
  • 评测方法本身是变量,建议榜单发布前用此框架自检可信度

多选题评测固定题目和正确答案,但未固定评测框架:选项顺序、提示词表述、模型答案来源(生成文本或选项似然)。现有研究仅报告整体分数波动,未分析波动集中于哪些题目,以及这些题目是否决定了模型间差距。本文将大模型评测框架视为独立变量,逐项解析其影响。构建“脆弱性网格”:12个来自4个系列的开源指令微调大模型,在4个基准(ARC、HellaSwag、MMLU、TruthfulQA)的3,679道题上,于26种合理评测配置下运行,每模型-题-配置组合记录一次正确性。所有模型权重与贪婪解码保持一致,仅变框架。结果显示,模型得分并非单一数值,而是一个区间——gemma4-31b得分在31%至89%之间波动。三项核心发现:在两个相邻模型均稳定回答的题目上,二者实际并列;而配置敏感题目平均贡献了95.7%的模型差距;12模型中有4个在特定配置下排名第一,说明赢家由评测方式决定。项目区分度(评测压缩方法追求的目标)与脆弱性相关系数为0.28(95%置信区间0.25–0.30),说明压缩保留了敏感题而非剔除。最关键的评分方式选择,远比选项顺序重要。作者公开逐题记录与分析脚本,所有数据可在秒级内复现,建议榜单发布前使用该网格进行可信性验证。

原文摘要 · Abstract (English)

Multiple-choice benchmarks fix the questions and the correct answers, but not the harness: the order of the options, the wording of the prompt, and whether a language model's answer is read from generated text or from per-option likelihoods. Work on this harness sensitivity reports it as aggregate score variance, leaving unexamined which items the variance falls on and whether they are the items that separate one model from the next. We treat the evaluation harness of large language models (LLMs) as an independent variable and resolve its effect to single items. We introduce the \textit{fragility grid}: 12 open-weight instruction-tuned LLMs from 4 families answer the same 3{,}679 items from 4 benchmarks (ARC, HellaSwag, MMLU, TruthfulQA) under 26 equally defensible harness configurations, recording one correctness bit for every model, item, and configuration. The comparison is matched, since the items, the weights, and the greedy decoding stay fixed while only the harness varies. Under the grid a model's score is a band rather than a point: gemma4-31b scores between 31 and 89 percent depending only on the harness. Three results follow. On the items that two adjacent models both answer stably the pair is tied, and config-fragile items carry 95.7 percent of a pair's gap on average. Four of the 12 models reach rank one under some configuration, so the harness selects the winner. Item discrimination, the property that benchmark-compression methods maximize, correlates with fragility at 0.28 (95 percent CI 0.25 to 0.30), so compression keeps the fragile items rather than removing them. The scoring choice, not the option order that protocols usually fix, is the load-bearing axis. We release the per-item records and the analysis script, from which every number regenerates on a CPU in seconds, and we position the fragility grid as a check a leaderboard can run before it reports an order.

大模型评测框架敏感性基准测试可复现性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。