单阶段训练让大模型轻松处理长文本，突破传统多阶段瓶颈。
Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models
- 用可调旋转位置编码统一训练，无需多阶段迭代
- 在4个基准上表现媲美甚至超越多阶段方法
- 适合需要高效长文本建模的开发者与研究者
大型语言模型（LLMs）在自然语言处理中取得突破，但受限于训练上下文长度，难以处理长序列，影响下游任务性能。现有长上下文建模方法多采用多阶段持续预训练，需大量人工调参与专家经验。本文提出一种新型单阶段持续预训练方法——头自适应旋转位置编码（HARPE），通过为不同注意力头设置不同旋转位置编码基频，在目标上下文长度下直接训练模型。在4个语言建模基准（包括最新RULER基准）上的实验表明，HARPE仅用单阶段训练即可有效理解并整合长上下文任务，性能达到甚至超过现有多阶段方法。结果证明，HARPE成功打破长上下文训练的阶段壁垒。
原文摘要 · Abstract (English)
Recently, Large language models (LLMs) have revolutionized Natural Language Processing (NLP). Pretrained LLMs, due to limited training context size, struggle with handling long token sequences, limiting their performance on various downstream tasks. Current solutions toward long context modeling often employ multi-stage continual pertaining, which progressively increases the effective context length through several continual pretraining stages. However, those approaches require extensive manual tuning and human expertise. In this paper, we introduce a novel single-stage continual pretraining method, Head-Adaptive Rotary Position Encoding (HARPE), to equip LLMs with long context modeling capabilities while simplifying the training process. Our HARPE leverages different Rotary Position Encoding (RoPE) base frequency values across different attention heads and directly trains LLMs on the target context length. Extensive experiments on 4 language modeling benchmarks, including the latest RULER benchmark, demonstrate that HARPE excels in understanding and integrating long-context tasks with single-stage training, matching and even outperforming existing multi-stage methods. Our results highlight that HARPE successfully breaks the stage barrier for training LLMs with long context modeling capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。