解决长文本输入下位置编码失效问题，提出几何优化的RoPE改进方法。
Frayed RoPE and Long Inputs: A Geometric Perspective
- 从几何角度分析注意力机制中键值分离特性，发现长输入破坏聚类结构。
- 提出RoPE-ID方法，通过高频旋转部分通道实现长序列即插即用泛化。
- 在1024以上长度文本上显著提升检索性能，适合长文本应用者。
旋转位置编码（RoPE）广泛用于语言模型的位置信息建模，但当输入长度超过训练长度时性能会下降。已有分析指出长输入导致通道旋转超出分布，但未说明其与异常行为的关联。本文通过实证与理论分析，提出统一的几何视角：注意力会诱导键和查询隐空间点云紧密聚类，形成可避免混淆的‘汇点’。长输入破坏这种分离，抑制汇点功能。基于此，我们提出RoPE-ID（In Distribution）——对部分通道施加高频率旋转，使注意力层天然支持更长输入。在10亿和30亿参数Transformer上，于LongBench和RULER信息检索任务中验证了该方法的有效性。
原文摘要 · Abstract (English)
Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models, which, while effective, causes performance breakdown when input length exceeds training length. Prior analyses assert (rightly) that long inputs cause channels to rotate ``out of distribution,'' but it is not clear how extra rotation relates to or causes pathological behavior. Through empirical and theoretical analysis we advance a unified geometric understanding of attention behavior with RoPE. We find that attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink tokens: placeholders that allow attention heads to avoid token mixing when not required. RoPE applied to longer inputs damages this key/query cluster separation, producing pathological behavior by inhibiting sink token functionality. From this geometric perspective, we propose RoPE-ID (In Distribution), a straightforward modification that allows attention layers to generalize to longer inputs out of the box: apply RoPE with high frequency to a subset of channels. We demonstrate the effectiveness of RoPE-ID for extended inputs using 1B and 3B parameter Transformers on the LongBench and RULER information retrieval benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。