模型自修正效果常被误读,格式修复可能伪装成推理改进。
The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale

- 通过控制语法解码,分离内容与格式影响
- 格式修复贡献超70%准确率变化,内容影响微弱
- 小模型自修正多为格式问题,大模型更需关注信号
语言模型自修正后的准确率变化通常被解释为推理改进。我们发现这一解释在答案提取边界上可能失效,并通过因果测试验证该问题。在Qwen3.5(0.8B-9B)、Gemma-4-12B及两个前沿模型(Tencent Hy3、Nvidia Nemotron-3-Ultra-550B)的29个主任务单元和一个前沿分支中,将始终修正的准确率变化分解为内容边际(双方可解析)与格式恢复/丢失边际(可解析性变化)。在12个具有显著不可解析答案率的单元中,格式效应超过内容效应(威尔科克森检验p=1.7e-3)。为因果验证,强制生成推理过程通过语法约束解码,使所有答案可解析:在14个单元中关闭了中位数71%的差距,两个单元完全收敛,最大效应单元仍有残差但未被忽略。聚类模型显示,0.8B/2B规模模型比能力级模型更易出现内容层面改变且有害(p<1e-7)。在Qwen3.5上复现引用的置信度门控协议未重现报告增益,且内容边际接近零。对更大模型的前沿检查显示,随规模增长格式主导性增强:尽管总效应达+0.275,5个单元内容边际均为零,但该分支功率较低。基于内容边际的校准下限标准揭示:低规模单元有空间但信号不足,能力级单元有信号但无空间;仅一个单元勉强可行,封闭保留收益极小。内容在领域测量的自修正中占少数份额。代码与工具已公开。
原文摘要 · Abstract (English)
Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning. We show this can fail at the answer-extraction boundary, and test the failure causally rather than only observationally. Across Qwen3.5 (0.8B-9B), Gemma-4-12B, and two frontier models via API (Tencent Hy3, Nvidia Nemotron-3-Ultra-550B) in 29 primary cells plus a frontier arm, we decompose the always-revise accuracy shift into a content margin (both answers parseable) and format-recovery/loss margins (parseability changes). On 12 cells with meaningful unparseable-answer rates, format effects exceed content effects (Wilcoxon p=1.7e-3). To test this causally, we force already-generated reasoning through grammar-constrained decoding so every answer is parseable by construction: across 14 cells this closes a median 71% of the gap between the naive total effect and the content-margin estimate, with two cells converging exactly and a residual on the two largest-effect cells reported rather than dismissed. A clustered model confirms floor-scale (0.8B/2B) models have far higher odds of content-level change and harm than capable-scale models (p<1e-7). Replicating a cited confidence-gating protocol verbatim on Qwen3.5 does not reproduce its reported gain and shows the same near-zero content margin. A frontier check on much larger models shows format-dominance intensifying with scale: content margin is exactly zero in all 5 cells despite total effects up to +0.275, though this arm is lower-powered. The calibration-floor criterion on the content margin reveals a squeeze: floor-scale cells have headroom but insufficient signal, capable-scale cells have signal but little headroom; only one cell is marginally viable, with negligible sealed-holdout gain. Content is a minority share of what the field has measured as self-correction. We release the instrument, code, and derived results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。