提出TAPA方法，让模型更好处理长文本中的位置信息。
Positional Encoding via Token-Aware Phase Attention
- 用可学习的相位函数替代固定位置编码，动态调整注意力
- 在长文本下困惑度更低，检索性能显著优于RoPE
- 无需微调即可扩展到更长序列，适合长文本任务
我们在实际假设下证明，旋转位置编码（RoPE）会在注意力分数中引入依赖距离的固有偏差，限制其对长上下文的建模能力。尽管已有扩展方法可缓解此问题，但通常需在预训练后进行后处理调整，如缩放或超参数重调。本文提出一种新型位置编码方法——令牌感知相位注意力（TAPA），将可学习的相位函数融入注意力机制。TAPA能有效保持长程令牌间交互，在直接且轻量的持续预训练下即可扩展至更长上下文，具备外推至未见长度的能力，并在长上下文场景下显著降低困惑度，提升检索性能，优于基于RoPE的基准方法。
原文摘要 · Abstract (English)
We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context. RoPE extension methods may alleviate this issue, but they typically require post-hoc adjustments after pretraining, such as rescaling or hyperparameters retuning. This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism. TAPA preserves token interactions over long range, extends to longer contexts with direct and light continual pretraining, extrapolates to unseen lengths, and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。