提出可扩展的无学习率训练方法，显著提升大模型预训练效果。
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
- 无需学习率调度，通过动态调整优化器参数实现稳定训练
- 在每参数1000词时性能超越当前最优调度方法31%
- 适用于长周期训练，支持预训练中模型平均与检查点合并
无学习率调度训练（Schedule-Free Learning）作为一种实用的即时训练方法，在数十个标准基准任务中表现出色。然而，其在大语言模型训练中的优异表现仅限于小规模场景。本文识别出将该方法扩展至更大批量和模型规模所需的关键改进，并提出一种无学习率、无调度的训练方法（ScheduleFree+），用于大语言模型训练，其性能远超温升-稳定-衰减（Warmup-Stable-Decay, WSD）调度方案。我们还证明，该方法在长时间训练中表现最佳，在每参数1000词的设定下，相比最先进调度方法性能提升31%。此外，该方法为预训练阶段的模型平均与检查点合并提供了理论依据。
原文摘要 · Abstract (English)
Schedule-Free Learning has shown promise as a practical anytime training method for machine learning, showing success across dozens of standard benchmark problems. However, strong performance for LLM training has only been demonstrated at small scales. We identify a number of fixes necessary to scale up Schedule-Free Learning to larger batch sizes and model sizes, and present a learning-rate-free and schedule-free method (ScheduleFree+) for training large language models which greatly outperforms Warmup-Stable-Decay (WSD) schedules. We also demonstrate that Schedule-Free Learning is most effective for long duration training, and at 1000 tokens per parameter, it outperforms SOTA schedules by 31%. Schedule-Free Learning provides a theoretical foundation for the use of model averaging and checkpoint merging during pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。