提出STRING方法，解决大模型实际使用时上下文长度不足的问题。
Why Does the Effective Context Length of LLMs Fall Short?
- 通过位置编码偏移，修复预训练中远距离信息获取能力缺陷。
- 无需额外训练，使Llama3.1 70B等模型在长文本任务上提升超10分。
- 适合需要高效利用长上下文的开源模型用户与研究者。
分布式训练和高效注意力机制的进步显著扩大了大语言模型（LLMs）的上下文窗口。然而，近期研究表明，开源LLM的有效上下文长度通常不足其训练长度的一半。本文将此限制归因于预训练与微调阶段相对位置的左偏分布，阻碍了远距离信息的有效聚合。为此，我们提出ShifTed Rotray position embeddING（STRING），在推理阶段将已训练好的位置编码进行偏移，覆盖原有无效位置，从而增强模型在现有训练长度内的表现。实验表明，无需额外训练，STRING显著提升了最新大模型如Llama3.1 70B和Qwen2 72B在RULER与InfiniteBench等主流长文本基准上的性能，提升超过10点，创下开源模型新纪录。相较商业模型，启用 extit{STRING}的Llama 3.1 70B表现优于GPT-4-128K，明显超越Claude 2与Kimi-chat。
原文摘要 · Abstract (English)
Advancements in distributed training and efficient attention mechanisms have significantly expanded the context window sizes of large language models (LLMs). However, recent work reveals that the effective context lengths of open-source LLMs often fall short, typically not exceeding half of their training lengths. In this work, we attribute this limitation to the left-skewed frequency distribution of relative positions formed in LLMs pretraining and post-training stages, which impedes their ability to effectively gather distant information. To address this challenge, we introduce ShifTed Rotray position embeddING (STRING). STRING shifts well-trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths. Experimental results show that without additional training, STRING dramatically improves the performance of the latest large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on popular long-context benchmarks RULER and InfiniteBench, establishing new state-of-the-art results for open-source LLMs. Compared to commercial models, Llama 3.1 70B with \method even achieves better performance than GPT-4-128K and clearly surpasses Claude 2 and Kimi-chat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。