用低秩稀疏分解拆解注意力机制，看清模型如何处理不同位置的特征交互。
Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
- 将Transformer注意力分解为可解释的低秩稀疏组件，分离出原始多头注意力的混合行为。
- 发现针对算术运算的专属注意力头，对应Llama-3.1-8B中的原子操作。
- 相比稀疏自编码器，该方法在电路发现上表现更优，适合研究模型内部机制。
我们提出低秩稀疏注意力（Lorsa），作为Transformer注意力层的稀疏替代模型，旨在将原始多头自注意力（MHSA）分解为独立可理解的组件。该方法解决注意力叠加问题，以揭示不同标记位置间特征的注意力介导交互。实验表明，Lorsa头部能发现更清晰、更细粒度的已有行为，如归纳头、后继头以及对首个标记高度关注的注意力汇聚现象。Lorsa与稀疏自编码器（SAE）同为稀疏字典学习方法，分别应用于不同Transformer组件，却在多个方面得出一致结论。例如，我们发现一个完整的算术专用Lorsa头部家族，每个对应于Llama-3.1-8B中的原子操作。自动可解释性分析显示，Lorsa在可解释性上达到与SAE相当水平，且在多头共同计算特征的电路发现上表现更优。我们还进行了广泛的架构消融实验、缩放定律研究及误差分析。
原文摘要 · Abstract (English)
We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components. Lorsa is designed to address the challenge of attention superposition to understand attention-mediated interaction between features in different token positions. We show that Lorsa heads find cleaner and finer-grained versions of previously discovered MHSA behaviors like induction heads, successor heads and attention sink behavior (i.e., heavily attending to the first token). Lorsa and Sparse Autoencoder (SAE) are both sparse dictionary learning methods applied to different Transformer components, and lead to consistent findings in many ways. For instance, we discover a comprehensive family of arithmetic-specific Lorsa heads, each corresponding to an atomic operation in Llama-3.1-8B. Automated interpretability analysis indicates that Lorsa achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties, especially for features computed collectively by multiple MHSA heads. We also conduct extensive experiments on architectural design ablation, Lorsa scaling law and error analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。