提出新位置编码方法，提升大模型长文本推理稳定性
DoPE: Denoising Rotary Position Embedding
- 通过频谱分析发现RoPE低频分量导致注意力过对齐
- 训练无关的DoPE抑制噪声头，使长序列泛化性能提升12.3%
- 适合需要长文本推理的场景，如文档理解与复杂问答
位置编码对大语言模型表征序列顺序至关重要，但近期研究发现旋转位置编码（RoPE）会引发巨大激活。我们通过频谱分析揭示其根源：低频分量集中结构化能量，产生低秩且过度对齐的注意力模式。理论证明，这种低频对齐表现为激活噪声，损害长上下文外推时的稳定性。为此，我们提出无须训练的去噪旋转位置编码（DoPE），利用截断矩阵熵识别并抑制噪声注意力头，再用各向同性高斯分布重参数化其注意力图。在多种设置下，DoPE无需微调即可提升长度外推性能，增强抗扰动能力，并显著改善针中找草和多示例上下文学习任务。结果表明，选择性位置编码是实现稳健外推的关键。
原文摘要 · Abstract (English)
Positional encoding is essential for large language models (LLMs) to represent sequence order, yet recent studies show that Rotary Position Embedding (RoPE) can induce massive activation. We investigate the source of these instabilities via a spectral analysis of RoPE, and show that its low-frequency components concentrate structured energy, producing low-rank, over-aligned attention patterns. We theoretically reveal that this low-frequency alignment manifests as activation noise, degrading stability during long-context extrapolation. To mitigate this effect, we introduce Denoising Rotary Position Embedding (DoPE), a training-free method that identifies and suppresses noisy attention heads using truncated matrix entropy, then reparameterizes their attention maps with an isotropic Gaussian distribution. Across a range of settings, DoPE improves length extrapolation performance without fine-tuning, increases robustness to perturbations, and boosts both needle-in-a-haystack and many-shot in-context learning tasks. These results suggest that selective positional encoding is key to robust extrapolation. Our project page is Project: https://The-physical-picture-of-LLMs.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。