用权重平均实现无需预知训练时长的高效语言模型训练
Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
- 通过权重平均与多项式衰减学习率，实现不依赖训练时长的随时可中断训练
- 在150M到300M参数模型上，达到与余弦调度相当的最终损失
- 适合持续训练场景，避免为固定算力反复调参
大语言模型越来越多地在持续或开放设置中训练，总训练时长无法预先确定。然而，现有预训练方法大多不具备随时性：依赖于依赖时长的学习率调度，并需在固定算力下进行大量调参。本文从理论上证明了过参数化线性回归存在随时学习率调度，且权重平均（又称模型合并）在实现随机梯度下降的极小极大收敛速率中起关键作用。我们发现这些随时调度随时间多项式衰减，衰减速率由问题的源条件和容量条件决定。实验评估了150M和300M参数的语言模型，在长达32倍Chinchilla规模的训练中，比较常数调度、$1/\sqrt{t}$ 调度与权重平均，以及经过优化的余弦调度。在整个训练范围内，随时调度达到与余弦调度相当的最终损失。结果表明，权重平均结合简单、无时长依赖的步长，为大语言模型预训练提供了一种实用有效的随时替代方案。
原文摘要 · Abstract (English)
Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance. Despite this, most existing pretraining recipes are not anytime: they rely on horizon-dependent learning rate schedules and extensive tuning under a fixed compute budget. In this work, we provide a theoretical analysis demonstrating the existence of anytime learning schedules for overparameterized linear regression, and we highlight the central role of weight averaging - also known as model merging - in achieving the minimax convergence rates of stochastic gradient descent. We show that these anytime schedules polynomially decay with time, with the decay rate determined by the source and capacity conditions of the problem. Empirically, we evaluate 150M and 300M parameter language models trained for up to 32x Chinchilla scale, comparing constant and $1/\sqrt{t}$ schedules with weight averaging against a well-tuned cosine schedule. Across the full training range, the anytime schedules achieve comparable final loss to cosine decay. Taken together, our results suggest that weight averaging combined with simple, horizon-free step sizes offers a practical and effective anytime alternative to cosine learning rate schedules for large language model pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。