拆解大模型算术外推失败的四个阶段,揭示根本原因。
Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs
- 通过最小GPT控制实验,分阶段分析外推失败机制。
- 3位数泛化失败源于布局、进位语义、重组与残余错误四阶段。
- 适合研究模型推理机制与外推缺陷的研究者参考。
算术基准常被简化为单一测试得分,但该得分可能混杂不同类型的失败。我们研究了一个在完整两位数加法数据上训练的极简GPT,其中所有局部数字转移均已覆盖,但仍无法泛化到三位数。失败过程是分阶段的:第一阶段为布局障碍——学习绝对位置的模型在纯三位布局变化下崩溃,仅混合布局暴露能显著削弱此障碍;第二阶段,在修复布局后,百位表现如进位标志而非语义数字,针对性进位探测可反转相关逻辑概率差,而额外数据对照组无效;第三阶段,修复进位后主要瓶颈为条件重构:高条件尾部数据在所有真实三位数任务中优于匹配对照组、仅高数据和仅尾部数据,且同一模式在更大2层桥接实验中重现;修复后残余错误主要集中在仅十位情况,10次种子的后期研究显示,引入符号感知的十位修复使最难的千位进位任务准确率从0.664提升至0.822。因此,我们提出了一个可实验验证的算术外推失败分解框架,包含布局、进位语义、重构与晚期十位残余四个阶段。
原文摘要 · Abstract (English)
Arithmetic benchmarks are often reduced to a single held-out score, but that score can conflate qualitatively different failures. We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails. The failure is staged. First, there is a layout barrier: a learned absolute-position model collapses under a pure 3-digit layout shift, and mixed-layout exposure is the only intervention that materially weakens this barrier. Second, after layout repair, the hundreds position behaves like a carry flag rather than a semantic hundreds digit; targeted carry probes reverse the relevant logit margin, whereas a matched extra-data control does not. Third, after carry repair, the main remaining bottleneck is conditional recomposition: high-conditioned tail data outperforms a matched control, high-only data, and tail-only data on all true-3-digit suites, and the same ordering reappears in a larger 2-layer bridge experiment. The residual errors after recomposition are then overwhelmingly tens-only, and a separate 10-seed late-stage study shows that a sign-aware tens repair raises exact match on the hardest thousands-carry suite from 0.664 to 0.822. We therefore provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。