发现模型性能在多数评测中由两个核心因子决定,可大幅减少评估成本。
You Don't Need to Run Every Eval
- 用秩-2矩阵补全方法从少量评测推断整体性能
- 5个基准测试可预测其余成绩,误差仅3.93分
- 提供可信度评估机制,适合预算有限的模型评估
当前模型发布通常报告40多个基准测试的成绩,且这些评估在训练过程中被重复运行以追踪进展、对比设计并选择最终检查点。但是否必须运行所有评测?我们构建了一个包含84个前沿模型在133个基准上的公开分数矩阵(2,604个单元格,填充率23.3%),发现其近似为秩-2:模型在各基准上的得分主要由两个数值决定。通过两种方式验证:(i) 隐蔽分数可由两个因子最佳恢复;(ii) 两个因子已解释共享基准上模型间90%以上的方差。基于此,我们提出BenchPress:一种对数空间的秩-2矩阵补全方法,能将缺失分数恢复至4.6分以内。使用BenchPress,我们发现五个基准{GPQA-D, HLE, Codeforces, MMLU-Pro, ARC-AGI-1}可将其余成绩恢复至3.93分误差内。若追求更低成本,{GPQA-D, MMLU-Pro, Aider Polyglot, MATH-500, AIME 2026}也可实现4.55分内预测。最后,我们识别出影响预测可靠性的因素,并结合预测分歧度量来判断何时可信赖预测结果。我们已将分数矩阵及实验代码开源至GitHub。
原文摘要 · Abstract (English)
A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release. But do we need to run every eval? We compile a public score matrix of 84 frontier models on 133 benchmarks (2,604 cells, 23.3% filled) and find it is approximately rank-2: a model's scores across all benchmarks are largely determined by just two numbers. We confirm this in two ways: (i) scores hidden from the matrix are best recovered using two factors, and (ii) two factors already explain over 90% of the variation among models on the benchmarks they share. Building on this, we design BenchPress: a logit-space rank-2 matrix completion method that recovers held-out scores to within 4.6 points. Using BenchPress, we find a subset of five benchmarks {GPQA-D, HLE, Codeforces, MMLU-Pro, ARC-AGI-1} that can recover the rest of a model's public scorecard to within 3.93 points. For a tighter evaluation budget, a cheaper set {GPQA-D, MMLU-Pro, Aider Polyglot, MATH-500, AIME 2026} can predict a model's evals to within 4.55. Finally, we identify what affects prediction reliability and combine these factors with predictor disagreement to quantify when predictions can be trusted. We release the score matrix, the code for reproducing all of our experiments on Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。