任务类型决定模型各层状态编码方式,而非仅由架构决定。
Task Structure Reverses Layerwise State Encoding in Sequence Models

- 同一模型在不同任务下,状态编码位置会反转:奇偶性任务中晚期集中,嵌套括号任务中早期集中。
- 微调后模型在特定任务中出现中间层瓶颈,如Pythia-160M在第6-7层缺失导致准确率下降81%。
- 状态可读性与计算瓶颈不一致,关键计算可能隐藏在非可读层中,需结合因果干预验证。
序列模型的层间状态编码常被视为架构特性:循环模型集中可读状态,注意力模型分布化。我们发现,当任务变化时,相同架构会反转这一模式。在Transformer、Mamba、Mamba-2、LSTM和GRU中,奇偶性任务的状态在Mamba和循环基线中集中在后期,在Transformer中则逐步构建;而在有界深度的Dyck-k任务中,模式翻转。微调后的Mamba-130M和Pythia-160M也呈现此现象,且Pythia的Dyck瓶颈在410M规模仍持续存在。文献中混淆了代数结构(交换性)与计算结构(前缀更新 vs. 栈机制)。通过引入非交换的S_3置换组合任务,发现所有五种架构在层间探查中均以奇偶性分组,而非Dyck,且在Mamba上使用Conv1D归因也支持此结论,说明分组依据为计算结构而非交换性。因果干预表明,在4层形式模型中,线性可读方向往往功能必要,且在分布外长度下仍重要。预训练模型则表现出分裂图景:微调后的Pythia在中间层存在强瓶颈(如160M时第6-7层消融导致准确率下降约81%,410M时更广范围的4-18层平台),而预训练的Mamba最终层高度可读,单一探针方向无法破坏奇偶性、Dyck或S_3任务,但中间激活修补可恢复97-98%的干净-损坏对数差距。探查仅定位线性可用状态,并非总对应计算瓶颈。机制特征是架构与任务共同决定的属性。
原文摘要 · Abstract (English)
Mechanistic studies of sequence models often treat layerwise state encodings as architectural traits: recurrent models concentrate readable state, attention-based models distribute it. We find that the same architecture reverses this profile when the task changes. Across Transformers, Mamba, Mamba-2, LSTMs, and GRUs, Parity is concentrated late in Mamba and the recurrent baselines and built gradually by Transformer; on bounded-depth Dyck-k the pattern flips. The same flip appears in fine-tuned Mamba-130M and Pythia-160M, and the Pythia Dyck bottleneck persists at 410M. Two explanations are conflated in the literature: algebraic structure (commutativity) versus computational structure (prefix update vs. stack). To separate them we add a third task: non-commutative S_3 permutation composition. S_3 groups with Parity, not Dyck, on layerwise probing across all five architectures and on Mamba-specific Conv1D attribution, so the grouping tracks computational structure rather than commutativity. Causal interventions show that, in the 4-layer formal models, linearly readable directions are often functionally necessary and can remain important at out-of-distribution lengths on Parity and Dyck. At pretrained scale the picture splits. Fine-tuned Pythia Dyck has a strong middle-layer bottleneck (L6-L7 ablation drops accuracy by roughly 81% at 160M; broader L4-L18 plateau at 410M), far weaker at the best-probe layer. Pretrained Mamba shows the complementary failure mode: its final layer is highly readable, no single probe direction breaks the task on Parity, Dyck, or S_3, yet mid-position activation patching there recovers about 97-98% of the clean-corrupted logit gap. Probing localizes where state is linearly available, not always where the computation is bottlenecked. Mechanistic signatures are properties of architecture and task together.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。