通过信息熵不变性提升注意力机制的长序列外推能力
Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms
- 基于信息熵不变性设计新缩放温度，无需训练
- 在64倍长输入下性能领先现有方法
- 揭示了注意力分数稀释是长序列处理的关键挑战
自2021年研究大语言模型长序列外推能力以来，部分方法修改了缩放点积注意力中的缩放因子，但缺乏理论依据。为此，本文提出两种基于信息熵不变性的新缩放温度以增强长度外推能力。首先，提出无需训练的InfoScale方法，通过保持熵一致，在长序列外推中维持对原始标记的关注；其次，理论上分析了缩放（CosScale）对余弦注意力的影响。实验表明，将InfoScale与CosScale结合，在GAU-α模型上实现64倍于训练长度的上下文窗口扩展，性能优于七种现有方法。分析显示，显著增大CosScale可近似窗口注意力，凸显注意力分数稀释是长程上下文处理的核心挑战。代码与数据见https://github.com/HT-NEKO/Information-Entropy-Invariance。
原文摘要 · Abstract (English)
Since the emergence of research on improving the length extrapolation capabilities of large language models in 2021, some studies have made modifications to the scaling factor in the scaled dot-product attention mechanism as part of their proposed methods without rigorous theoretical justifications. To fill this gap, we propose two new scaled temperatures based on information entropy invariance to enhance length extrapolation. First, a training-free method InfoScale is designed for dotproduct attention, and preserves focus on original tokens during length extrapolation by ensuring consistent entropy. Second, we theoretically analyze the impact of scaling (CosScale) on cosine attention. Experimental data demonstrates that combining InfoScale and CosScale achieves state-ofthe-art performance on the GAU-α model with a context window extended to 64 times the training length, and outperforms seven existing methods. Our analysis reveals that significantly increasing CosScale approximates the Windowed Attention, and highlights the significance of attention score dilution as a key challenge in long-range context handling. The code and data are available at https://github.com/HT-NEKO/ Information-Entropy-Invariance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。