用累积变换提升注意力模型长序列外推能力，揭示其原理与局限
Why Do Accumulated Transformations Extrapolate?
- 用累积的旋转替代位置编码，实现更优的外推性能
- 累积变换在长序列时因信号抑制而性能下降，存在理论极限
- 适合研究注意力机制外推性、长序列建模的学者参考
PaTH Attention发现，将RoPE的位置索引旋转替换为累积的数据相关Householder反射，可实现强长度外推，但在极长上下文时性能下降。本文探讨这种现象是否依赖Householder结构，还是累积变换的一般特性。研究简化版本：保留RoPE的块对角SO(2)旋转，但将位置索引角度替换为累积的标记依赖角度，仍呈现相同模式——外推提升后在长上下文出现退化。理论上证明，满足一定正则性条件的累积正交变换，其乘积在有限步后趋于无相干性，抑制远距离标记注意力。累积查询与键的旋转形成与上下文长度无关的有限混合窗口；训练中学习的每标记抑制机制可无损迁移至任意评估长度；高维集中效应产生得分差距，压制远距离标记，而近程路径传输保留目标信号。反向下界表明，累积旋转最终必退化：当远端集合扩大，不显式控制远端质量，无法同时保持近端信号。对于SO(2)旋转，仅旋转值会导致远端残差贡献无相干叠加，扩展影响范围。受控实验验证预测：随机累积旋转显著优于RoPE，学习的标记依赖旋转在训练长度之外维持低困惑度，且旋转值有助于查询和键单独旋转。仅旋转模型在极端长度仍退化，而ALiBi保持长度稳定，符合需远端质量控制的结论。
原文摘要 · Abstract (English)
PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths. We ask whether this depends on Householder-specific structure or reflects a general property of accumulated transformations along source-to-query paths. We study a simpler variant keeping RoPE's block-diagonal SO(2) rotations but replacing position-indexed angles with accumulated token-dependent ones. It shows the same pattern: improved extrapolation then degradation at long contexts. We prove the result extends to accumulated orthogonal transformations satisfying certain regularity conditions: their products become incoherent after finitely many steps, suppressing attention to distant tokens. Accumulated rotations of queries and keys create a finite mixing window independent of context length; per-token suppression learned in training transfers unchanged to any evaluation length, and high-dimensional concentration produces a score gap suppressing far tokens while near-route transport preserves the target signal. Conversely, a lower bound shows accumulated rotations must eventually degrade: as the far set grows, no rotations preserve the near signal without explicit far-mass control. For SO(2) rotations, rotating values too makes residual far contributions combine incoherently, extending the range. Controlled experiments support these predictions: random accumulated rotations substantially improve extrapolation over RoPE, learned token-dependent rotations maintain near-training-length perplexity far beyond the training context, and rotating values helps over queries and keys alone. Rotation-only models still degrade at extreme lengths, while ALiBi stays length-stable, consistent with the need for far-mass control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。