提出可自适应旋转角度的位置编码，提升模型对长序列的建模能力。
Selective Rotary Position Embedding
- 基于输入动态调整旋转角度，统一适配 softmax 与线性注意力机制。
- 在复制、状态追踪等任务上显著提升性能，优于传统 RoPE。
- 揭示了隐式位置结构，为位置编码设计提供新思路，适合序列建模研究者。
位置信息对语言建模至关重要。在 softmax 变换器中，旋转位置编码（RoPE）通过固定角度旋转编码位置；而在线性变换器中，顺序由依赖输入的（选择性）门控处理，衰减过去的键值关联。选择性通常被证明能提升语言任务表现。受此启发，我们提出选择性旋转位置编码（Selective RoPE），一种输入依赖的旋转编码机制，可泛化 RoPE 并支持任意角度旋转，适用于 softmax 与线性变换器。我们发现，softmax 注意力本身对查询-键对就隐含执行了这类旋转，揭示了隐式的位置结构。进一步地，在状态空间模型和门控线性变换器中，实部负责遗忘，虚部通过旋转编码位置。通过为门控变换器引入 Selective RoPE，我们在语言建模及复制、状态追踪、检索等复杂序列任务上验证了其有效性，输入依赖的旋转显著提升了性能。
原文摘要 · Abstract (English)
Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we introduce \textit{Selective RoPE}, an \textit{input-dependent} rotary embedding mechanism, that generalizes \textit{RoPE}, and enables rotation in \textit{arbitrary angles} for both linear and softmax transformers. We show that softmax attention already performs a hidden form of these rotations on query-key pairs, uncovering an implicit positional structure. We further show that in state-space models and gated linear transformers, the real part manages forgetting while the imaginary part encodes positions through rotations. We validate our method by equipping gated transformers with \textit{Selective RoPE}, demonstrating that its input-dependent rotations improve performance in language modeling and on difficult sequence tasks like copying, state tracking, and retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。