用模型合并替代学习率衰减，提升大模型预训练效果
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
- 通过合并训练检查点模拟不同学习率衰减策略
- 在MATH、HumanEval等任务上分别提升3.5%~5.5%
- 适合追求高效预训练与长期模型优化的研究者
近期学习率调度进展表明，无衰减方法可在不降低性能的前提下取代传统衰减阶段。模型合并技术成为该领域极具前景的解决方案。本文提出温升-稳定-合并（WSM）框架，建立了学习率衰减与模型合并之间的形式化联系。WSM将多种衰减策略——包括余弦衰减、线性衰减和反平方根衰减——统一为有理论依据的模型平均方案，且兼容多种优化方法。大量实验表明，合并时长（检查点聚合的训练窗口）是影响模型性能最关键的因子，优于检查点间隔与合并数量。该框架在多个基准测试中持续超越广泛使用的温升-稳定-衰减（WSD）方法，在MATH上提升+3.5%，HumanEval上+2.9%，MMLU-Pro上+5.5%。性能优势在监督微调场景中同样显著，凸显其在长期模型精炼中的潜力。
原文摘要 · Abstract (English)
Recent advances in learning rate (LR) scheduling have demonstrated the effectiveness of decay-free approaches that eliminate the traditional decay phase while maintaining competitive performance. Model merging techniques have emerged as particularly promising solutions in this domain. We present Warmup-Stable and Merge (WSM), a general framework that establishes a formal connection between learning rate decay and model merging. WSM provides a unified theoretical foundation for emulating various decay strategies-including cosine decay, linear decay and inverse square root decay-as principled model averaging schemes, while remaining fully compatible with diverse optimization methods. Through extensive experiments, we identify merge duration-the training window for checkpoint aggregation-as the most critical factor influencing model performance, surpassing the importance of both checkpoint interval and merge quantity. Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks, achieving significant improvements of +3.5% on MATH, +2.9% on HumanEval, and +5.5% on MMLU-Pro. The performance advantages extend to supervised fine-tuning scenarios, highlighting WSM's potential for long-term model refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。