仅用局部键就能识别关键长序列注意力头，提升效率。
Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
- 用局部键的二阶矩近似预测长上下文注意力头
- 部分注意力头在长序列中表现关键，可被精准识别
- 适合关注长文本建模与注意力机制优化的研究者
长上下文处理对自然语言处理任务至关重要，但仍是重大挑战。尽管注意力机制效率已大幅提升，其在长上下文中的运作机制仍不清晰。本文发现：某些注意力头始终只关注局部信息，而另一些则随查询在局部与长上下文间动态切换。这引发疑问：能否仅凭局部键判断哪些头对准确预测下一个词至关重要？我们证明，仅使用局部键即可有效预测关键长上下文注意力头。核心思想是通过二阶矩近似建立长上下文得分的简化模型。该发现揭示了长序列中注意力的简单特性，为显著提升效率提供了可能。
原文摘要 · Abstract (English)
The ability to process long contexts is crucial for many natural language processing tasks, yet it remains a significant challenge. While substantial progress has been made in enhancing the efficiency of attention mechanisms, there is still a gap in understanding how attention heads function in long-context settings. In this paper, we observe that while certain heads consistently attend to local information only, others swing between attending to local and long-context information depending on the query. This raises the question: can we identify which heads require long-context information to predict the next token accurately? We demonstrate that it's possible to predict which heads are crucial for long-context processing using only local keys. The core idea here is to exploit a simple model for the long-context scores via second moment approximations. These findings unveil simple properties of attention in the context of long sequences, and open the door to potentially significant gains in efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。