arXiv:2605.22488cs.LG2026-05

用算术任务检验Transformer是否真的按预设步骤计算,发现模型虽表示中间结果但未真正使用。

Represented Is Not Computed: A Causal Test of Candidate Algorithmic Intermediates in a Transformer

  • 通过精确的算术任务设计,识别出模型可能的中间计算步骤。
  • 模型准确率达99.83%,且线性探针能解码中间变量。
  • 因果测试显示中间结果被表示但未传到输出,揭示探针结论与真实路径差异。

结构化提示需要根据任务相关的关联整合组件。在语言或视觉任务中,这种关联通常不够明确,难以定义候选内部算法。算术提供更清晰的设置。我们研究了一个在基础位提取任务上训练的Transformer:给定N、B和D,需报告N在基-B展开中B^D项的系数。闭式解为⌊N/B^D⌋ mod B,提供了明确的候选算法中间步骤。在三个随机种子下,模型在未见的数-基交叉组合上达到99.83%的精确答案准确率,证明其具备可靠的任务能力。线性探针可解码这些中间步骤,表明分阶段算术计算是可能的。因果测试进一步将表示与使用分离:在从输入D到输出位置的局部路径中,行为仅依赖早期对D的选择性通信,与N和B无关。此外,稀疏电路搜索发现,主要存在独立的N、B、D路径,它们在后期才结合,而非探针所暗示的分阶段路径。因此,模型虽然表示了使闭式解合理的中间步骤,但识别出的局部因果路径并未将这些中间结果传递至输出流。这一案例表明,即使有明确的算法假设,基于探针的结论也可能与因果观察严重偏离。

原文摘要 · Abstract (English)

Structured prompts require integrating components according to task-relevant relations. How a network implements this integration is often hard to judge in language or vision, where those relations are rarely specified precisely enough to define a candidate internal algorithm. Arithmetic offers a cleaner setting. We study a Transformer trained on base-digit extraction: given $N$, $B$, and $D$, it must report the coefficient of $B^D$ in the base-$B$ expansion of $N$. The closed-form solution, $\lfloor N/B^D \rfloor \bmod B$, provides explicit candidate algorithmic intermediates. Across three seeds, the model reaches 99.83% exact-answer accuracy on held-out number-base intersections, establishing reliable task competence. Linear probes decode the intermediates, making staged arithmetic computation plausible. Causal tests then separate representation from use: within the localized route from the stream with $D$ as input to the output positions, behavior depends on early $D$-selective communication, independent of $N$ and $B$. Relatedly, a sparse circuit search finds mostly separate $N$, $B$, and $D$ routes that combine late rather than the staged route suggested by the probes. Thus, the model represents the intermediates that make the closed-form solution plausible, but the identified localized causal route does not transmit them to the output stream. This case shows that probe-based conclusions can diverge sharply from causal observations, even when explicit algorithmic hypotheses are available.

Transformer因果分析算法推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。