分析小模型计数时注意力头是协作还是竞争。
Do Attention Heads Compete or Cooperate during Counting?
- 注意力头像伪集成，共同完成计数子任务。
- 输出需非均匀聚合才能符合语法结构。
- 揭示了注意力机制在基础推理中的协同模式。
我们对小型Transformer模型在基础计数任务上的训练进行了深入的机制可解释性分析，该任务是许多算法中关键的推断步骤。具体而言，我们研究了注意力头之间的协作/竞争关系：注意力头是否表现为伪集成，共同解决同一子任务，或各自承担不同子任务，仅通过协作才能完成原任务。研究结果表明，在计数任务语义层面，注意力头表现如伪集成，但其输出需以非均匀方式聚合，才能生成符合语法的编码。源代码将在发表后公开。
原文摘要 · Abstract (English)
We present an in-depth mechanistic interpretability analysis of training small transformers on an elementary task, counting, which is a crucial deductive step in many algorithms. In particular, we investigate the collaboration/competition among the attention heads: we ask whether the attention heads behave as a pseudo-ensemble, all solving the same subtask, or they perform different subtasks, meaning that they can only solve the original task in conjunction. Our work presents evidence that on the semantics of the counting task, attention heads behave as a pseudo-ensemble, but their outputs need to be aggregated in a non-uniform manner in order to create an encoding that conforms to the syntax. Our source code will be available upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。