提出nD-RoPE，让Transformer更好处理高维数据的位置信息。
nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding
- 将位置与频率耦合为高维向量，实现无分解的统一位置编码
- 在图像、视频、点云任务中均提升性能，方向无关性更强
- 适合需要高维建模的视觉与几何学习任务
旋转位置编码（RoPE）广泛应用于Transformer模型，但其在高维场景的扩展缺乏统一理论。现有方法多沿各轴独立旋转或经验混合频率，限制了跨维度交互，导致表示依赖方向。为此，我们提出nD-RoPE，一种无需分解的任意维度广义RoPE。基于连续希尔伯特空间中的平移不变性，推导出保持各向同性的谱条件，要求将位置与频率视为耦合的n维向量。通过多尺度正单纯形波矢设计实现该形式，提供非退化空间覆盖及对称、方向平衡的二阶响应。在图像、视频和点云任务上的实验表明，该方法在高维设置中持续提升性能并增强泛化能力。
原文摘要 · Abstract (English)
Rotary Position Embedding (RoPE) is widely adopted in Transformer models, yet its extension to high-dimensional domains lacks a unified theoretical formulation. Most existing approaches either apply rotations independently along each axis or empirically mix frequencies, which limits cross-dimensional interactions and yields direction-dependent representations. To address these limitations, we propose nD-RoPE, a decomposition-free generalization of RoPE to arbitrary dimensions. From a translation-invariant formulation in continuous Hilbert space, we derive a spectral condition for isotropy that requires treating positions and frequencies as coupled \(n\)-dimensional vectors. We instantiate this formulation with a multi-scale regular-simplex wave-vector design, which provides non-degenerate spatial coverage and a symmetric, directionally balanced second-order response. Experiments across images, videos, and point clouds demonstrate consistent performance gains and improved generalization in high-dimensional settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。