发现Transformer反向传播中隐含的注意力机制，可无损修改模型行为。
Reversed Attention: On The Gradient Descent Of Attention Layers In GPT
- 提出反向注意力机制，揭示梯度传播中的隐含计算
- 通过注意力修补实现不改权重的前向行为调控
- 提升模型可解释性，适合关注模型内部机制的研究者
基于Transformer的语言模型成功得益于其注意力机制。尽管该机制在可解释性研究中已被广泛探讨，主要聚焦于前向传播中获得的注意力值，但注意力的反向传播却长期被忽视。本文研究了注意力反向传播的数学本质，发现其隐式计算出一个我们称为「反向注意力」的注意力矩阵。我们分析了反向注意力的性质，并证明其能有效揭示模型行为与编辑动态。在实验中，我们展示了一种名为「注意力修补」的新方法，可直接操纵前向注意力过程，而无需修改模型权重。反向注意力不仅深化了对语言模型在反向传播中如何配置注意力层的理解，还使反向传播过程更具可解释性。
原文摘要 · Abstract (English)
The success of Transformer-based Language Models (LMs) stems from their attention mechanism. While this mechanism has been extensively studied in explainability research, particularly through the attention values obtained during the forward pass of LMs, the backward pass of attention has been largely overlooked. In this work, we study the mathematics of the backward pass of attention, revealing that it implicitly calculates an attention matrix we refer to as "Reversed Attention". We examine the properties of Reversed Attention and demonstrate its ability to elucidate the models' behavior and edit dynamics. In an experimental setup, we showcase the ability of Reversed Attention to directly alter the forward pass of attention, without modifying the model's weights, using a novel method called "attention patching". In addition to enhancing the comprehension of how LM configure attention layers during backpropagation, Reversed Attention maps contribute to a more interpretable backward pass.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。