研究发现，变压器模型虽能解码层级信息，但未必真正使用这些信息。
Dissociating Decodability and Causal Use in Bracket-Sequence Transformers
- 通过探测和干预残差流与注意力模式，分析层次结构表示
- 屏蔽正确栈顶位置导致长距离准确率骤降，而残差流删减影响小
- 揭示可解码不等于因果使用，适用于语言建模与形式语言研究
在训练于需要理解层次结构的任务时，变压器模型会以不同方式表示这种结构：体现在残差流的几何形态中，以及保持后进先出顺序的栈式注意力模式中。然而，这些表示是否被真正用于推理尚不明确。本文研究了在平衡括号序列（Dyck语言）上的变压器模型，其层次结构有明确真值。通过探测和干预残差流与注意力模式，发现深度、距离和栈顶信号均可解码，但其因果作用不同：屏蔽真实栈顶位置会导致长距离准确率显著下降，而删减低维残差流子空间的影响则较小。该结果在模板化自然语言场景中也成立，表明即使在已知相关层次变量的受控设置下，可解码性并不等同于因果使用。
原文摘要 · Abstract (English)
When trained on tasks requiring an understanding of hierarchical structure, transformers have been found to represent this hierarchy in distinct ways: in the geometry of the residual stream, and in stack-like attention patterns maintaining a last-in, first-out ordering. However, it remains unclear whether these representations are causally used or merely decodable. We examine this gap in transformers trained on the Dyck language (a formal language of balanced bracket sequences), where the hierarchical ground truth is explicit. By probing and intervening on the residual stream and attention patterns, we find that depth, distance, and top-of-stack signals are all decodable, yet their causal roles diverge. Specifically, masking attention to the true top-of-stack position causes a sharp drop in long-distance accuracy, while ablating low-dimensional residual stream subspaces has comparatively little effect. These results, which extend to a templated natural language setting, suggest that even in a controlled setting where the relevant hierarchical variables are known, decodability alone does not imply causal use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。