让大模型长文本能力翻倍，且短文本表现几乎不变
LongRoPE2: Near-Lossless LLM Context Window Scaling
- 通过进化搜索优化位置编码，解决长文本训练不足问题
- 128K上下文长度下，短文本性能保留超98.5%
- 仅需100亿token，远少于其他方法，适合实际部署
LongRoPE2是一种新方法，可将预训练大语言模型的有效上下文窗口扩展至目标长度，同时保持原始短上下文性能。核心贡献包括：(1) 提出假设，即高维位置编码训练不足是现有方法持续出现分布外（OOD）问题的原因；(2) 设计一种基于“针刺式”困惑度引导的进化搜索算法，有效解决训练不足问题；(3) 采用混合上下文训练策略，在微调中使用重缩放的位置编码处理长序列，同时保留原始位置编码对短序列的性能。在LLaMA3-8B和Phi3-mini-3.8B上跨多个基准的实验证明了该假设并验证了方法有效性。显著的是，LongRoPE2将LLaMA3-8B扩展至128K有效上下文长度，短文本性能保留超过98.5%，仅使用100亿标记——比Meta的方法少80倍，而后者未能达到目标长度。
原文摘要 · Abstract (English)
LongRoPE2 is a novel approach that extends the effective context window of pre-trained large language models (LLMs) to the target length, while preserving the performance on the original shorter context window. This is achieved by three contributions: (1) a hypothesis that insufficient training in higher RoPE dimensions contributes to the persistent out-of-distribution (OOD) issues observed in existing methods; (2) an effective RoPE rescaling algorithm that adopts evolutionary search guided by "needle-driven" perplexity to address the insufficient training problem; (3) a mixed context window training approach that fine-tunes model weights to adopt rescaled RoPE for long-context sequences while preserving the short-context performance with the original RoPE. Extensive experiments on LLaMA3-8B and Phi3-mini-3.8B across various benchmarks validate the hypothesis and demonstrate the effectiveness of LongRoPE2. Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context length while retaining over 98.5% of short-context performance, using only 10B tokens -- 80x fewer than Meta's approach, which fails to reach the target effective context length. Code will be available at https://github.com/microsoft/LongRoPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。