模型基准提升未必代表能力增强，关键要看答案是否真正实现。
Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains
- 区分答案'可达性'与'实现性'，用固定预算审计每道题表现。
- 随机路由在43种设置中不弱于结构化搜索，但需知道正确答案才能见效。
- 关闭特定MLP模块可修复68%~92%的失败案例，揭示关键瓶颈。
基准分数提升常被视为模型能力增强的证据，但相同提升可能源于不同行为变化：模型可能首次获得正确答案，或仅是原本就可达的答案被输出。聚合分数无法区分两者。本文在固定预算、温度和答案格式下建立问题级审计机制。当默认部署流程产生正确答案时，称该问题被‘实现’；当指定探测器在固定预算内找到答案，则称其‘可达’。实验发现，在43个模型与任务组合中，随机路由在相同预算下表现不逊于结构化搜索，但无答案感知的程序几乎无法保留此增益，说明该优势依赖对正确答案的访问。进一步分析显示，在六组0.5B至31B规模的案例中，抑制特定MLP块可修复68%至92%的预定义失败集。再测试训练是否通过扩大可达性来提升性能：在六个匹配评估中的五组，部署表现上升，而可达上限却持平或下降；以DAPO为例，部署得分提升14.7分，但可达上限反而下降13.3分。因此，在审计设置中，实现与可达性并不总同步变化。声称能力扩展应同时报告实际表现与匹配条件下的可达性。代码已开源。
原文摘要 · Abstract (English)
Benchmark gains are often treated as evidence of greater LLM capability. Yet the same gain can reflect different changes in model behavior. A model may reach new answers, or produce answers that were already within reach. Aggregate scores do not distinguish these changes question by question. We establish a question-level audit under fixed budgets, temperatures, and answer formats. A question is realized when the default deployment procedure produces the correct answer. A question is reachable when a specified probe finds that answer within a fixed budget. We first test whether inference-time layer routing can expand reachability. Under a matched budget, random routes match or exceed structured search in all 43 model and task settings. Answer-blind procedures retain almost none of this gain, which instead requires access to the correct answer. We then ask why reachable answers sometimes fail to appear. Across six cases spanning 0.5B to 31B, silencing one identified MLP block repairs 68 to 92 percent of a predefined failure set. We next test whether training closes the gap by expanding reachability. In five of six matched evaluations, deployed performance rises while the reachable ceiling remains flat or falls. For DAPO, the deployed score rises by 14.7 points while the reachable ceiling falls by 13.3 points. Across the settings we audit, realization and reachability therefore do not always change together. Claims of capability expansion should report both realized performance and reachability under matched evaluation conditions. Code is available at https://github.com/LiZaiyuan0619/reachability-not-realization
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。