arXiv:2605.10414cs.LG2026-05

让注意力更懂取舍,长文本理解更准。

Remember to Forget: Gated Adaptive Positional Encoding

论文配图:Remember to Forget: Gated Adaptive Positional Encoding
图 1 · 摘自论文原文
  • 用查询和键控制的门控机制,动态筛选无关上下文
  • 在长文本中保持关键信息可访问,远距离注意力衰减更合理
  • 无需改造模型结构,直接提升旋转位置编码表现

旋转位置编码(RoPE)广泛用于现代大语言模型。当序列长度超过训练时范围时,旋转相位会进入分布外状态,导致虚假长程对齐、注意力弥散和检索性能下降。现有方法仅部分缓解此问题,常以牺牲局部定位精度为代价换取长程稳定性。本文提出GAPE(门控自适应位置编码),一种可直接嵌入位置编码的增强方案,在保留旋转几何结构的同时,将内容感知偏差引入注意力分数。GAPE通过查询依赖门控收缩无关上下文,通过键依赖门控保留重要远距离令牌。我们证明受保护令牌始终可访问,而未受保护远距离令牌的注意力质量随查询门控函数衰减。此外,GAPE可在标准缩放点积注意力中实现。实验验证其在合成检索与长上下文基准上均显著优于旋转基线,带来更清晰的注意力分布与更强的长程鲁棒性。

原文摘要 · Abstract (English)

Rotary Positional Encoding (RoPE) is widely used in modern large language models. However, when sequences are extended beyond the range seen during training, rotary phases can enter out-of-distribution regimes, leading to spurious long-range alignments, diffuse attention, and degraded retrieval. Existing remedies only partially address these failures, as they often trade local positional resolution for long-context stability. We propose GAPE (Gated Adaptive Positional Encoding), a drop-in augmentation for positional encodings that introduces a content-aware bias directly into the attention logits while preserving the rotary geometry. GAPE decouples distance-based suppression from token importance through a query-dependent gate that contracts irrelevant context and a key-dependent gate that preserves salient distant tokens. We prove that protected tokens remain accessible, while the attention mass assigned to unprotected distant tokens decays as a function of the query gate. We further show that GAPE can be implemented within standard scaled dot-product attention. We validate these properties empirically, finding that GAPE consistently yields sharper attention and improved long-context robustness over rotary baselines across both synthetic retrieval and long-context benchmarks.

位置编码注意力机制长文本理解大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。