提出旋转位置编码的相位调制理论,给出长上下文模型的参数设计边界。
Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
- 将RoPE视为复振荡器的相位调制,用信号处理理论分析其行为
- 推导出防止位置混淆与相位漂移的基底参数下限,随深度增加而收紧
- 发现浮点精度限制导致参数上限,超出则位置信息完全丢失
旋转位置编码(RoPE)通过乘法旋转编码词元位置,广泛用于大语言模型,但其在长上下文中的表现仍缺乏系统理解。本文将其重新解释为复振荡器集合上的相位调制,借助经典信号处理理论进行分析。在此框架下,推导出保障位置一致性的基底参数下界:包括类奈奎斯特的混叠限制,以及抑制低频位置模式相位漂移的直流分量稳定性约束。进一步扩展至深层模型,发现多层旋转调制会累积角度错位,使基底要求随深度增强。同时,基于有限浮点精度,推导出基底参数的精度相关上限:超过该值后,相位增量无法区分,导致位置信息彻底丧失,即使无混叠亦然。上下界共同定义了依赖精度与深度的可行性区域,即长上下文模型的‘恰到好处’区间。通过LLaMA、Mistral及DeepSeek等前沿模型的案例研究验证,模型成功、失败与社区修复均与预测边界高度吻合。违反稳定性边界者出现注意力坍缩与长程退化;尝试扩展至百万级上下文时,遭遇独立于架构与训练的硬性精度壁垒。
原文摘要 · Abstract (English)
Rotary positional embeddings (RoPE) are widely used in large language models to encode token positions through multiplicative rotations, yet their behavior at long context lengths remains poorly characterized. In this work, we reinterpret RoPE as phase modulation applied to a bank of complex oscillators, enabling analysis through classical signal processing theory. Under this formulation, we derive principled lower bounds on the RoPE base parameter that are necessary to preserve positional coherence over a target context length. These include a fundamental aliasing bound, analogous to a Nyquist limit, and a DC-component stability bound that constrains phase drift in low-frequency positional modes. We further extend this analysis to deep transformers, showing that repeated rotary modulation across layers compounds angular misalignment, tightening the base requirement as depth increases. Complementing these results, we derive a precision-dependent upper bound on the RoPE base arising from finite floating-point resolution. Beyond this limit, incremental phase updates become numerically indistinguishable, leading to positional erasure even in the absence of aliasing. Together, the lower and upper bounds define a precision- and depth-dependent feasibility region a Goldilocks zone for long-context transformers. We validate the framework through a comprehensive case study of state-of-the-art models, including LLaMA, Mistral, and DeepSeek variants, showing that observed successes, failures, and community retrofits align closely with the predicted bounds. Notably, models that violate the stability bound exhibit attention collapse and long-range degradation, while attempts to scale beyond one million tokens encounter a hard precision wall independent of architecture or training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。