研究LLaMA-3如何在路由后通过最后一位确定三位数加法结果
Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions
- 发现第17层后仅依赖最后一个输入位和自注意力
- 数字方向可通过低秩正交映射在共享空间中关联
- 旋转方向可精准编辑结果，适合模型可解释性研究
我们在单标记输出设定下研究Meta-Llama-3-8B模型的三位数加法，以刻画跨标记路由失效后的答案形成机制。因果残差修补与累计注意力消融实验定位到约第17层存在清晰边界：此后解码结果几乎完全由最后一个输入标记控制，晚期自注意力基本可忽略。在此后路由阶段，数字（和）方向随更高一位上下文变化，但其关系可通过一个近似正交映射在共享低秩子空间内统一建模（低秩Procrustes对齐）。因果数字编辑验证了该几何结构：简单跨上下文迁移失败，而通过学习映射旋转方向可恢复严格反事实编辑；负向对照则无法复现。
原文摘要 · Abstract (English)
We study three-digit addition in Meta-Llama-3-8B (base) under a one-token readout to characterize how arithmetic answers are finalized after cross-token routing becomes causally irrelevant. Causal residual patching and cumulative attention ablations localize a sharp boundary near layer~17: beyond it, the decoded sum is controlled almost entirely by the last input token and late-layer self-attention is largely dispensable. In this post-routing regime, digit(-sum) direction dictionaries vary with a next-higher-digit context but are well-related by an approximately orthogonal map inside a shared low-rank subspace (low-rank Procrustes alignment). Causal digit editing matches this geometry: naive cross-context transfer fails, while rotating directions through the learned map restores strict counterfactual edits; negative controls do not recover.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。