通过对比协方差分解注意力空间，揭示模型关注特定词元的内在机制。
Decomposing Query-Key Feature Interactions Using Contrastive Covariances
- 提出对比协方差方法，将查询-键空间分解为低秩可解释分量。
- 发现查询与键在这些子空间对齐时，注意力得分显著升高。
- 适用于大语言模型中语义类别和绑定特征的可解释性分析。
尽管注意力头在Transformer中扮演核心角色，我们仍缺乏理解模型为何关注某个词元的工具。为此，我们研究查询-键（QK）空间——即查询与键之间的双线性联合嵌入空间。本文提出一种对比协方差方法，将QK空间分解为低秩、人类可解释的分量。当查询与键在这些低秩子空间中对齐时，会产生高注意力分数。我们首先在简化设置下进行理论与实证分析。随后将该方法应用于大型语言模型，识别出用于分类语义特征和绑定特征的人类可解释的QK子空间。最后，展示了如何将注意力分数归因于所识别的特征。
原文摘要 · Abstract (English)
Despite the central role of attention heads in Transformers, we lack tools to understand why a model attends to a particular token. To address this, we study the query-key (QK) space -- the bilinear joint embedding space between queries and keys. We present a contrastive covariance method to decompose the QK space into low-rank, human-interpretable components. It is when features in keys and queries align in these low-rank subspaces that high attention scores are produced. We first study our method both analytically and empirically in a simplified setting. We then apply our method to large language models to identify human-interpretable QK subspaces for categorical semantic features and binding features. Finally, we demonstrate how attention scores can be attributed to our identified features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。