语言模型准确率掩盖了失败状态，新框架分离执行与判断
Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets
- 提出双层评估框架，区分执行状态与正确性判断
- 450个Qwen输出中49个未生成最终答案，远高于DeepSeek的5个
- 适合评估测试时方法、需关注执行过程而非仅准确率
语言模型基准测试将两个独立问题合并为单一准确率：响应是否进入可评估状态，以及答案是否正确。我们提出一种双层评估框架，将评分无关的执行证据（如终止状态、答案暴露、可解析性、完成长度）与评分依赖的正确性分离。在五种固定Qwen和DeepSeek配置下，对MATH和ARC-Challenge数据集的2,550个输出进行测试，匹配2,048令牌限制时，执行模式差异显著：450个Qwen MATH输出中有49个未生成最终答案，而300个DeepSeek MATH输出仅5个缺失，750个ARC输出无一缺失。在相同300个DeepSeek MATH样本中，当令牌上限提升至8,192时，未生成最终答案的情况消失。覆盖审计的针对性验证研究进一步表明，候选选择与聚合策略会显著改变对比准确率估计。结果表明，准确率混淆了执行案例分布与验证策略。因此，测试时方法的评估应同时报告干预前的执行状态、验证覆盖率及评分来源。
原文摘要 · Abstract (English)
Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。