揭示大模型算术错误背后的几何结构，解释为何加法常出错。
The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models
- 发现加法计算中存在锚定语义数字的连续进位轨迹。
- 算术错误源于神经噪声使进位潜势越过量化阈值。
- 该几何框架可检测并修正推理中的算术错误，适合模型调试者。
大型语言模型在基础算术上表现出矛盾的脆弱性，暗示其内部计算与离散输出之间存在脱节。通过分析多操作数加法过程中的残差流几何结构，我们识别出一种名为等原始和轨迹（IRST）的几何模式：表征由语义数字锚定，并由连续的进位纤维调制。我们提出噪声量化模型来解释这一几何结构，将算术错误归因于内部神经噪声推动连续的潜在进位值跨越量化阈值所导致的几何滑移。该几何框架还进一步阐明了探针的多功能性，解释了轻量级探针如何从单个激活向量中解耦共存的潜在信号（如真实值与幻觉）。最后，我们通过一种几何一致性检查方法验证了这些见解，该方法能在推理过程中有效检测并纠正此类量化失败。代码已公开于 https://github.com/RL-MIND/Shape-of-Addition。
原文摘要 · Abstract (English)
Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a geometric structure where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model to explain this geometry, framing arithmetic errors as Geometric Slippages caused by internal neural noise pushing a continuous, latent Carry Potential across quantization thresholds. This geometric framework further elucidates Probe Versatility, explaining how lightweight probes can disentangle coexisting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference. Our code is available at https://github.com/RL-MIND/Shape-of-Addition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。