用反周期位置编码让模型在上下文中精准找信息，解决检索不稳定问题。
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
- 设计反周期频率梯度，使序列首尾确定性耦合，实现可靠定位
- 512上下文下检索准确率达90.3%，远超传统方法的63.3%
- 无需额外训练成本，仅换频率表就能提升检索稳定性
Möbius RoPE 是一种基于反周期频率梯度 θ_i=π(2i+1)/N 的旋转位置编码：每个旋转平面在训练上下文中前进奇数倍 π，使位置全纯性为 -1，序列两端通过闭式狄利克雷“偶极子”确定性耦合；据我们所知，这是首个在位置编码中引入反周期边界条件的工作。我们数值验证理论精度达 ~10⁻⁶，并预训练了48个模型，涵盖六组160M级和三组410M级分支（每组2B FineWeb-Edu token；混合分支将莫比乌斯频率用于25%的注意力头）。混合模型困惑度无变化（29.66 vs. 29.72），但针堆检索变得可靠：512上下文下准确率为90.3±5.7% 对比 63.3±31.4%（n=6种子），最差种子表现86% 对比 14%，稳健性检验 p=0.013–0.029（未校正），410M规模也重现（Levene p=0.040）。对照实验表明，同一频段的非周期梯度无效果，周期性（全纯性 +1）梯度仅部分有效。将训练后模型的频率表回换至标准RoPE（权重冻结）导致检索崩溃，损伤集中于远端针尖；训练模型依赖此长程几何结构。无位置编码（NoPE）分支在短上下文更可靠，但困惑度上升13%且外推性能最差；唯有反周期混合分支同时保持基线困惑度与高可靠性下限。该效应限定于训练窗口内的单针检索；一次频率表替换即可提供零成本保险，规避检索种子随机性风险。
原文摘要 · Abstract (English)
Möbius RoPE is a rotary positional encoding built on the anti-periodic frequency ladder $θ_i=π(2i+1)/N$: every rotation plane advances by an odd multiple of $π$ across the training context, so the positional holonomy is $-1$ and the two ends of the sequence are deterministically coupled through a closed-form Dirichlet "dipole"; to our knowledge this is the first anti-periodic boundary condition in positional encoding. We verify the theory numerically to $\sim 10^{-6}$ and pretrain 48 models spanning six 160M-class and three 410M-class arms (2B FineWeb-Edu tokens each; the hybrid arm puts Möbius frequencies on 25% of heads). Hybrid perplexity is unchanged (29.66 vs. 29.72), but needle-in-a-haystack retrieval becomes reliable: $90.3\pm5.7\%$ versus $63.3\pm31.4\%$ at context 512 ($n=6$ seeds), observed worst seed 86% versus 14%, robust variance tests $p=0.013$-$0.029$ (unadjusted), recurring at 410M (Levene $p=0.040$). Matched controls isolate the mechanism: an aperiodic ladder in the same frequency band reproduces none of the effect, and a periodic (holonomy $+1$) ladder only a fraction. Swapping trained models' frequency table back to standard RoPE (weights frozen) collapses retrieval, with damage concentrated on far needles: trained models depend on this long-range geometry. A NoPE arm is even more reliable at short context but pays a 13% perplexity tax and extrapolates worst; only the anti-periodic hybrid pairs baseline perplexity with a high reliability floor. The effect is scoped to single-needle retrieval within the training window; a one-line frequency swap thus provides zero-cost insurance against the retrieval seed lottery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。