用算术任务检验Transformer是否真的按预设步骤计算,发现模型虽表示中间结果但未真正使用。
Represented Is Not Computed: A Causal Test of Candidate Algorithmic Intermediates in a Transformer
- 通过精确的算术任务设计,识别出模型可能的中间计算步骤。
- 模型准确率达99.83%,且线性探针能解码中间变量。
- 因果测试显示中间结果被表示但未传到输出,揭示探针结论与真实路径差异。
结构化提示需要根据任务相关的关联整合组件。在语言或视觉任务中,这种关联通常不够明确,难以定义候选内部算法。算术提供更清晰的设置。我们研究了一个在基础位提取任务上训练的Transformer:给定N、B和D,需报告N在基-B展开中B^D项的系数。闭式解为⌊N/B^D⌋ mod B,提供了明确的候选算法中间步骤。在三个随机种子下,模型在未见的数-基交叉组合上达到99.83%的精确答案准确率,证明其具备可靠的任务能力。线性探针可解码这些中间步骤,表明分阶段算术计算是可能的。因果测试进一步将表示与使用分离:在从输入D到输出位置的局部路径中,行为仅依赖早期对D的选择性通信,与N和B无关。此外,稀疏电路搜索发现,主要存在独立的N、B、D路径,它们在后期才结合,而非探针所暗示的分阶段路径。因此,模型虽然表示了使闭式解合理的中间步骤,但识别出的局部因果路径并未将这些中间结果传递至输出流。这一案例表明,即使有明确的算法假设,基于探针的结论也可能与因果观察严重偏离。
原文摘要 · Abstract (English)
Structured prompts require integrating components according to task-relevant relations. How a network implements this integration is often hard to judge in language or vision, where those relations are rarely specified precisely enough to define a candidate internal algorithm. Arithmetic offers a cleaner setting. We study a Transformer trained on base-digit extraction: given $N$, $B$, and $D$, it must report the coefficient of $B^D$ in the base-$B$ expansion of $N$. The closed-form solution, $\lfloor N/B^D \rfloor \bmod B$, provides explicit candidate algorithmic intermediates. Across three seeds, the model reaches 99.83% exact-answer accuracy on held-out number-base intersections, establishing reliable task competence. Linear probes decode the intermediates, making staged arithmetic computation plausible. Causal tests then separate representation from use: within the localized route from the stream with $D$ as input to the output positions, behavior depends on early $D$-selective communication, independent of $N$ and $B$. Relatedly, a sparse circuit search finds mostly separate $N$, $B$, and $D$ routes that combine late rather than the staged route suggested by the probes. Thus, the model represents the intermediates that make the closed-form solution plausible, but the identified localized causal route does not transmit them to the output stream. This case shows that probe-based conclusions can diverge sharply from causal observations, even when explicit algorithmic hypotheses are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。