改进位置编码，让模型更擅长处理长文本
Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
- 用傅里叶级数构建位置编码，抑制有害频率分量
- 在不同上下文长度下保持更稳定性能，提升长文本泛化能力
- 适合需要长序列建模的NLP任务，如文档理解
通过改进旋转位置编码（RoPE）来扩展语言模型的上下文长度已成为趋势。本文发现，RoPE的局限性不仅存在于注意力机制内部，还影响了语言模型几乎所有部分。基于离散信号处理理论，我们揭示了RoPE通过隐式实现非均匀离散傅里叶变换来实现周期性注意力，但该周期性会因线性层和激活函数引起的频谱损伤，以及时域截断带来的频率分量训练不足而被破坏。为此，我们提出傅里叶位置编码（FoPE），通过构建傅里叶级数并剔除有害频率成分，增强注意力的频域特性，从而改善周期性外推与长度泛化能力。在多种模型规模和基准测试中，相比其他基线方法，FoPE在不同上下文窗口下均表现出更稳定的性能。多项分析与消融实验进一步验证了方法的有效性和理论建模的合理性。
原文摘要 · Abstract (English)
Extending the context length of Language Models (LMs) by improving Rotary Position Embedding (RoPE) has become a trend. While prior works mainly address RoPE's limitations within attention, this paper uncovers the adverse effects on length generalization from nearly all parts of LMs. Using Discrete Signal Processing theory, we show that RoPE enables periodic attention by implicitly achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is undermined by the spectrum damage caused by: 1) linear layers and activation functions; 2) insufficiently trained frequency components brought by time-domain truncation. Building on our observations, we propose Fourier Position Embedding (FoPE), which enhances attention's frequency-domain properties to improve both its periodic extension and length generalization. FoPE constructs \textit{Fourier Series} and zero-outs the destructive frequency components, increasing model robustness against the spectrum damage. Experiments across various model scales and benchmarks show that, within varying context windows, FoPE maintains a more stable performance compared to other baselines. Several analyses and ablations bring further support to our method and theoretical modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。