提升位置编码性能，让Transformer在长文本中更精准捕捉位置关系。
Higher-Dimensional Rotary Position Embedding
- 用高维旋转替代传统2D旋转，增强位置编码的表达能力。
- 在多个基准上显著超越标准RoPE，长/短上下文均表现更优。
- 无需额外参数，工程实现高效，适合部署到实际系统中。
Transformer在长序列建模中普遍依赖位置编码机制。旋转位置编码（RoPE）通过独立的二维旋转嵌入位置信息，在自注意力中形成相对位置项。然而，其成对、分块且解耦的结构限制了通道间的深度混合与鲁棒性。本文提出HD-RoPE，将RoPE从独立的2D旋转扩展至高维旋转，并引入Paley-I正交基，实现每个旋转子空间内的均衡、各向同性与密集相位混合。这显著增强了通道耦合性和旋转自由度，同时保持正交稳定性与相对位置闭包性质。此外，HD-RoPE可高效优化，无需引入额外可训练参数。大量实验表明，HD-RoPE在多个主流基准上均显著优于标准RoPE，无论在长上下文还是短上下文场景下均有明显提升。
原文摘要 · Abstract (English)
Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention. However, its pairwise, block-based, and decoupled structure limits deep mixing and robustness across channels. We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace. This significantly enhances channel coupling and rotational degrees of freedom while maintaining orthogonal stability and the relative position closure property. Furthermore, HD-RoPE is easily optimized for engineering efficiency without introducing additional trainable parameters. We have conducted extensive evaluation results demonstrating that HD-RoPE achieves significant performance improvements over standard RoPE across various popular benchmarks and in both long and short contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。