通过短到长的上下文窗口调度，提升大模型预训练效率与性能。
SkyLadder: Better and Faster Pretraining via Context Window Scheduling
- 先用短上下文预训练，再逐步扩展至长上下文。
- 在100B token上训练1B/3B模型，性能提升最高达3.7%。
- 训练速度最快快22%，适合追求高效预训练的研究者。
近期大语言模型预训练趋向于不断扩展上下文窗口以处理更长序列。然而我们的初步研究发现，在固定令牌预算下，使用较短上下文窗口预训练的模型始终优于长上下文版本。这一现象促使我们探索最优的上下文窗口调度策略，以更好平衡长上下文能力与预训练效率。为此，我们提出SkyLadder，一种简单而有效的方法，实现从短到长的上下文窗口渐进过渡。SkyLadder在保持优异标准基准表现的同时，于长上下文任务上达到或超过基线表现。通过大量实验，我们在100B tokens上预训练了1B参数模型（最长32K上下文）和3B参数模型（8K上下文），结果表明，SkyLadder在常见基准上持续获得最高3.7%的性能提升，同时相比基线实现最高22%的训练加速。代码已公开于https://github.com/sail-sg/SkyLadder。
原文摘要 · Abstract (English)
Recent advancements in LLM pretraining have featured ever-expanding context windows to process longer sequences. However, our pilot study reveals that models pretrained with shorter context windows consistently outperform their long-context counterparts under a fixed token budget. This finding motivates us to explore an optimal context window scheduling strategy to better balance long-context capability with pretraining efficiency. To this end, we propose SkyLadder, a simple yet effective approach that implements a short-to-long context window transition. SkyLadder preserves strong standard benchmark performance, while matching or exceeding baseline results on long context tasks. Through extensive experiments, we pre-train 1B-parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating that SkyLadder yields consistent gains of up to 3.7% on common benchmarks, while achieving up to 22% faster training speeds compared to baselines. The code is at https://github.com/sail-sg/SkyLadder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。