提出高维位置编码的数学框架，统一解释并扩展旋转位置嵌入
Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding
- 基于李群代数理论构建高维旋转位置编码的数学基础
- 证明轴对齐块对角结构对应最大环形子代数，且可学习正交变换调节跨维度交互
- 为图像等高维任务提供可解释的编码设计范式，适合多模态模型研究者
旋转位置编码（RoPE）因其高效的相对位置编码与强外推能力，被广泛应用于大语言模型。然而，在二维图像等更高维输入域中的应用虽已有尝试，仍缺乏统一的理论框架。为此，我们基于李群与李代数理论，提出了一个系统性的数学框架。通过分析RoPE的两个核心性质——相对性与可逆性，我们推导出任意有效N维RoPE的充要条件。研究表明，RoPE可被视为特殊正交李代数中极大交换子代数（MASA）的一组基，而常见的轴对齐块对角形式（每维独立使用2×2旋转块）对应于最大环形子代数。此外，我们发现空间跨维度交互可通过学习正交变换实现基底转换来简化。实验表明，跨维度交互需与局部结构保持平衡。本框架统一了现有RoPE设计，并为高维模态与任务提供了可遵循的扩展路径。
原文摘要 · Abstract (English)
Rotary Position Embedding (RoPE) is widely adopted in large language models (LLMs) due to its efficient encoding of relative positions with strong extrapolation capabilities. However, while its application in higher-dimensional input domains, such as 2D images, have been explored in several attempts, a unified theoretical framework is still lacking. To address this, we propose a systematic mathematical framework for RoPE grounded in Lie group and Lie algebra theory. We derive the necessary and sufficient conditions for any valid $N$-dimensional RoPE based on two core properties of RoPE - relativity and reversibility. We demonstrate that RoPE can be characterized as a basis of a maximal abelian subalgebra (MASA) in the special orthogonal Lie algebra, and that the commonly used axis-aligned block-diagonal RoPE, where each input axis is encoded by an independent 2x2 rotation block, corresponds to the maximal toral subalgebra. Furthermore, we reduce spatial inter-dimensional interactions to a change of basis, resolved by learning an orthogonal transformation. Our experiment results suggest that inter-dimensional interactions should be balanced with local structure preservation. Overall, our framework unifies and explains existing RoPE designs while enabling principled extensions to higher-dimensional modalities and tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。