arXiv:2508.02308cs.CL2025-08ACL被引 1

不训练即可自适应扩展长文本处理能力,解决大模型超长输入性能下降问题。

LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training

  • 基于动态缩放的分粒度位置编码,随输入长度自动调整位置信息分配
  • 在五个主流长文本基准上均显著超越现有方法,最长支持2048倍上下文扩展
  • 无需微调,可直接适配各类基于RoPE的大型语言模型

大型语言模型在输入超出预训练上下文窗口时会出现性能显著下降,主要源于旋转位置编码(RoPE)的分布外(OOD)行为。现有方法通过固定映射策略将OOD位置重映射到分布内范围,忽略了输入长度与模型有效上下文窗口之间的动态关系。为此,我们提出长度感知的多粒度位置编码(LaMPE),一种无需训练的方法,可充分使用模型的有效上下文窗口实现自适应长上下文扩展。受相对位置频率分布左偏的启发,LaMPE通过参数化缩放的Sigmoid函数建立映射长度与输入长度间的动态关系,实现不同输入长度下的位置容量自适应分配。同时,设计新颖的多粒度注意力机制,战略性地在序列不同区域分配不同粒度的位置分辨率,以同时捕捉细粒度局部性与长程依赖。该方法可无缝应用于多种基于RoPE的LLM,无需训练。在三个代表性模型上的五项主流长上下文基准测试表明,相比现有长度外推方法,LaMPE取得显著性能提升。代码将开源于https://github.com/scar-on/LaMPE。

原文摘要 · Abstract (English)

Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window, primarily due to the out-of-distribution (OOD) behavior of Rotary Position Embedding (RoPE). Recent studies mitigate this problem by remapping OOD positions into the in-distribution range with fixed mapping strategies, ignoring the dynamic relationship between input length and the model's effective context window. To this end, we propose Length-aware Multi-grained Positional Encoding (LaMPE), a training-free method that fully utilizes the model's effective context window for adaptive long-context scaling in LLMs. Motivated by the left-skewed frequency distribution of relative positions, LaMPE establishes a dynamic relationship between mapping length and input length through a parametric scaled sigmoid function to adaptively allocate positional capacity across varying input lengths. Meanwhile, LaMPE devises a novel multi-grained attention mechanism that strategically allocates positional resolution across different sequence regions to capture both fine-grained locality and long-range dependencies. Our method can be seamlessly applied to a wide range of RoPE-based LLMs without training. Extensive experiments on three representative LLMs across five mainstream long-context benchmarks demonstrate that LaMPE achieves significant performance improvements compared to existing length extrapolation methods. The code will be released at https://github.com/scar-on/LaMPE.

位置编码长文本无训练RoPE

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。