研究旋转位置编码在注意力机制中的动态行为,揭示其稳定状态与能量分布关系。
Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere

- 通过连续时间模型分析旋转位置编码对注意力动态的影响
- 发现共识态保持平衡,且存在明确的半角收缩边界和尾部衰减规律
- 适用于理解大模型中位置编码与注意力稳定性的内在关联
旋转位置编码(RoPE)通过位置相关的旋转改变注意力分数,但其对归一化标记动态的影响无法由传统的球面自注意力模型捕捉。本文研究当查询和键被旋转而值保持在单位球面上时的连续时间动力学。所得注意力核具有可逆性,并存在尖锐的均匀Softmax下界。尽管在固定非平凡系统内,自然的RoPE交互能量导数符号交替出现,每个共识态仍为平衡点,其横向线性化是依赖于共识点在各RoPE平面能量的可逆马尔可夫算子。在单频共振环上,推导出精确的贝塞尔混叠谱,涵盖非互质频率及正确的固定环大β渐近行为。全局上,闭半球不变;成对非钝角构型与严格开半圆均以显式半角和单点尾部界收缩。这些区域估计实现了基于尖锐RoPE Softmax下界的核泛化正性原理。RoPE还选择了一条显式的分数平坦化扭曲分支:通用共振族非双曲且线性不稳定,而奇数反向族在商去全局旋转后变为双曲鞍点。在多维情况下,局部共识差距可能随不同频率平面能量分配呈现非单调变化,因此不存在统一的频率排序。矩阵、有限差分与非线性流计算交叉验证了定理边界及报告常数。
原文摘要 · Abstract (English)
Rotary position embeddings (RoPE) modify attention scores through position-dependent rotations, but their effect on normalized token dynamics is not captured by the vanilla spherical self-attention model. We study the continuous-time dynamics obtained when queries and keys are rotated while values remain on the unit sphere. The resulting attention kernel is reversible and admits a sharp uniform softmax floor, yet the natural RoPE interaction energy has derivatives of both signs within one fixed nontrivial system. Every consensus state remains an equilibrium, and its transverse linearization is a reversible Markov operator whose kernel depends on the consensus point through its energy across RoPE planes. On a resonant single-frequency ring we derive an exact Bessel-aliasing spectrum, including non-coprime frequencies and the correct fixed-ring large-$β$ asymptotics. Globally, closed hemispheres are invariant, while pairwise non-obtuse configurations and strict open semicircles contract with explicit half-angle and single-point tail bounds. These regional estimates instantiate a kernel-generic positivity principle with the sharp RoPE softmax floor. RoPE also selects an explicit score-flattening twisted branch; the generic resonant family is non-hyperbolic and linearly unstable, whereas an odd antipodal family becomes a hyperbolic saddle after quotienting global rotation. In multiple dimensions, the local consensus gap can depend non-monotonically on the allocation of energy across frequency planes, so no universal ordering by frequency is valid. Independent matrix, finite-difference, and nonlinear-flow computations cross-check the theorem boundaries and the reported constants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。