arXiv:2605.14589cs.CL2026-05

用短序列训练实现长上下文扩展,效率远超传统方法。

EndPrompt: Efficient Long-Context Extension via Terminal Anchoring

论文配图:EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
图 1 · 摘自论文原文
  • 用两段式结构:原短文本+末尾提示,模拟长距离位置关系。
  • 在8K扩展到64K时,平均RULER得分达76.03,超越多个基线方法。
  • 适合资源有限但需长上下文能力的研究者和开发者使用。

扩展大语言模型的上下文窗口通常需要在目标长度序列上进行训练,带来二次方级的内存与计算开销,使长上下文适配成本高昂且难以复现。本文提出EndPrompt,仅用短训练序列即可实现有效上下文扩展。核心思想是:无需构建完整长序列,只需保留原始短上下文作为第一段,将简短的终端提示作为第二段,并赋予其接近目标上下文长度的位置索引。该两段式结构在短物理序列中引入局部与长程相对位置距离,同时保持训练文本语义连续性——这是基于分块模拟方法所缺失的特性。我们基于旋转位置编码(Rotary Position Embedding)与伯恩斯坦不等式提供理论分析,表明位置插值对注意力函数施加严格光滑性约束,共享的Transformer参数进一步抑制对未观测中间距离的不稳定外推。将该方法应用于LLaMA系列模型,从8K扩展至64K上下文窗口,平均RULER得分为76.03,在LongBench上表现最佳,优于LCEG(72.24)、LongLoRA(72.95)及全序列微调(69.23),且计算开销显著更低。结果表明,通过稀疏位置监督即可诱导长上下文泛化,挑战了需密集长序列训练才能可靠扩展上下文窗口的主流假设。代码已开源:https://github.com/clx1415926/EndPrompt。

原文摘要 · Abstract (English)

Extending the context window of large language models typically requires training on sequences at the target length, incurring quadratic memory and computational costs that make long-context adaptation expensive and difficult to reproduce. We propose EndPrompt, a method that achieves effective context extension using only short training sequences. The core insight is that exposing a model to long-range relative positional distances does not require constructing full-length inputs: we preserve the original short context as an intact first segment and append a brief terminal prompt as a second segment, assigning it positional indices near the target context length. This two-segment construction introduces both local and long-range relative distances within a short physical sequence while maintaining the semantic continuity of the training text--a property absent in chunk-based simulation approaches that split contiguous context. We provide a theoretical analysis grounded in Rotary Position Embedding and the Bernstein inequality, showing that position interpolation induces a rigorous smoothness constraint over the attention function, with shared Transformer parameters further suppressing unstable extrapolation to unobserved intermediate distances. Applied to LLaMA-family models extending the context window from 8K to 64K, EndPrompt achieves an average RULER score of 76.03 and the highest average on LongBench, surpassing LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23) while requiring substantially less computation. These results demonstrate that long-context generalization can be induced from sparse positional supervision, challenging the prevailing assumption that dense long-sequence training is necessary for reliable context-window extension. The code is available at https://github.com/clx1415926/EndPrompt.

长上下文高效训练位置编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。