arXiv:2605.27980cs.CLcs.AI2026-05被引 2

提出周期性位置编码,让大模型突破长文本理解瓶颈。

Periodic RoPE for Infinite Context LLMs

  • 用周期性RoPE+滑动窗口捕捉局部依赖,避免位置耗尽。
  • 引入无位置编码全局层,实现全序列无限交互。
  • 实测模型在超长文本上更稳定高效,适合长文档处理。

处理超长上下文对大语言模型完成长时序任务至关重要。尽管近期工作已将上下文窗口扩展至100万以上,但当序列长度超过位置编码(如RoPE)的预训练范围时,模型性能会下降,即出现位置耗尽问题。这一根本限制必须克服,才能实现真正的无限上下文。为此,我们提出周期性RoPE(P-RoPE),一种可规避位置耗尽的位置编码机制。它与滑动窗口注意力(SWA)结合,捕捉每个窗口内的局部依赖和相对位置;再通过无位置编码(NoPE)的全局注意力层,实现整个序列无位置约束的无限交互。通过堆叠这两类层,模型无需位置外推即可泛化更长序列,理论上支持无限上下文窗口。实验表明,我们的模型MiniWin在长上下文效率与稳定性上优于标准GPT架构的MiniMInd。本工作为实现真正无限上下文理解的大模型提供了可行路径。代码见:https://github.com/Cominder/miniwin。

原文摘要 · Abstract (English)

The ability to process ultra-long contexts is crucial for large language models (LLMs) to perform long-horizon tasks. While recent efforts have extended context windows to 1M and beyond, model performance degrades when sequence length exceeds the pre-trained range of positional encodings (e.g., RoPE), i.e., position exhaustion. This fundamental limitation must be overcome to achieve a truly infinite context. To address it, we propose Periodic RoPE (P-RoPE), a positional encoding mechanism designed to circumvent this exhaustion. It operates in conjunction with sliding window attention (SWA) to capture local dependencies and relative positions within each window. This local layer is then complemented by a global attention layer with No Positional Encoding (NoPE), enabling unbounded interaction across the entire sequence without positional constraints. By stacking these two types of layers, the model avoids the need for positional extrapolation to generalize longer and theoretically supports an infinite context window. Empirical results show that our model, MiniWin, outperforms MiniMInd with standard GPT architectures in long-context efficiency and stability. Our work provides a possible pathway toward LLMs with genuine infinite-context understanding. The code is available at \href{https://github.com/Cominder/miniwin}{https://github.com/Cominder/miniwin}.

位置编码长文本无限上下文注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。