让旋转位置编码更完整，提升长文本理解能力
Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
- 用复数的虚部重构注意力分数，保留更多位置关系信息
- 在长文本任务中显著提升性能，越长文本优势越明显
- 适合需要精准长距离依赖建模的场景，如法律/医疗文本分析
旋转位置编码（RoPE）已成为大语言模型中编码序列顺序的标准方法，通过在复平面中对查询和键向量进行旋转实现。然而，标准实现仅使用复数点积的实部计算注意力分数，忽略了包含重要相位信息的虚部，可能导致长程依赖关系建模的信息损失。本文提出一种扩展方法，重新引入被丢弃的虚部信息，利用完整的复数表示构建双组件注意力分数。理论与实验均表明，该方法通过保留更多位置信息，显著增强了长程依赖建模能力。在一系列长上下文语言建模基准上的评估显示，该方法持续优于标准RoPE，且随着上下文长度增加，性能提升愈发显著。代码已开源：https://github.com/OpenMOSS/rope_pp。
原文摘要 · Abstract (English)
Rotary Position Embeddings (RoPE) have become a standard for encoding sequence order in Large Language Models (LLMs) by applying rotations to query and key vectors in the complex plane. Standard implementations, however, utilize only the real component of the complex-valued dot product for attention score calculation. This simplification discards the imaginary component, which contains valuable phase information, leading to a potential loss of relational details crucial for modeling long-context dependencies. In this paper, we propose an extension that re-incorporates this discarded imaginary component. Our method leverages the full complex-valued representation to create a dual-component attention score. We theoretically and empirically demonstrate that this approach enhances the modeling of long-context dependencies by preserving more positional information. Furthermore, evaluations on a suite of long-context language modeling benchmarks show that our method consistently improves performance over the standard RoPE, with the benefits becoming more significant as context length increases. The code is available at https://github.com/OpenMOSS/rope_pp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。