发现旋转位置编码中负责长距离依赖的关键注意力头
On the token distance modeling ability of higher RoPE attention dimension
- 按维度分析旋转位置编码，找出专司长距离信息的注意力头
- 这些头在长文本处理中起关键作用，删掉后性能显著下降
- 为提升模型上下文长度提供了可解释的新思路
基于旋转位置编码（RoPE）的长度外推算法在扩展语言模型上下文长度方面表现出色。然而，理解位置编码如何捕捉远距离上下文信息仍不清晰。我们基于不同维度对应不同变化频率的直觉，开展维度级分析，研究注意力头某一隐藏维度与其对长距离依赖建模能力的相关性。通过提出相关性度量，我们在多个长度外推模型中识别出一类特殊注意力头，命名为位置头（Positional Heads）。这些头高度聚焦于长距离信息交互，在长输入处理中起核心作用，实验证明其移除会导致性能明显下降。进一步表明，长度外推效率与这些头在高维注意力分配上的扩展程度密切相关。该发现为未来长文本理解研究提供了重要启示。
原文摘要 · Abstract (English)
Length extrapolation algorithms based on Rotary position embedding (RoPE) have shown promising results in extending the context length of language models. However, understanding how position embedding can capture longer-range contextual information remains elusive. Based on the intuition that different dimensions correspond to different frequency of changes in RoPE encoding, we conducted a dimension-level analysis to investigate the correlation between a hidden dimension of an attention head and its contribution to capturing long-distance dependencies. Using our correlation metric, we identified a particular type of attention heads, which we named Positional Heads, from various length-extrapolated models. These heads exhibit a strong focus on long-range information interaction and play a pivotal role in long input processing, as evidence by our ablation. We further demonstrate the correlation between the efficiency of length extrapolation and the extension of the high-dimensional attention allocation of these heads. The identification of Positional Heads provides insights for future research in long-text comprehension.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。