不用学习率衰减预训练，能让大模型下游微调表现更好
Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
- 预训练时用恒定学习率（WSO），不进行衰减
- 在1B和8B模型上，微调后性能优于传统衰减方法
- 适合关注模型泛化能力与下游适配性的研究者
我们研究了大规模语言模型预训练中学习率调度的作用，重点关注其对监督微调（SFT）下游性能的影响。虽然基于衰减的学习率调度广泛用于降低预训练损失，但其对微调后性能的影响仍缺乏深入探索。本文提出温启动后保持恒定学习率的Warmup-Stable-Only（WSO）策略。在1B和8B参数模型上的实验表明，尽管衰减策略在预训练阶段表现更优，但WSO在微调后性能上始终更佳，且该结论在中等训练和过拟合情况下均成立。损失曲面分析显示，衰减策略使模型进入更尖锐的极小值，而WSO则维持更平坦的极小值，有利于模型适应性。结果表明，为提升预训练指标而使用学习率衰减可能损害下游可适应性。本研究为训练和模型发布提供了实用建议：采用WSO预训练可增强模型对下游任务的适应能力。
原文摘要 · Abstract (English)
We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised fine-tuning (SFT). Decay-based learning rate schedulers are widely used to minimize pre-training loss. However, despite their widespread use, how these schedulers affect performance after SFT remains underexplored. In this paper, we examine Warmup-Stable-Only (WSO), which maintains a constant learning rate after warmup without any decay. Through experiments with 1B and 8B parameter models, we show that WSO consistently outperforms decay-based schedulers in terms of performance after SFT, even though decay-based schedulers may exhibit better performance after pre-training. The result also holds across different regimes with mid-training and over-training. Loss landscape analysis further reveals that decay-based schedulers lead models into sharper minima, whereas WSO preserves flatter minima that support adaptability. These findings indicate that applying LR decay to improve pre-training metrics may compromise downstream adaptability. Our work also provides practical guidance for training and model release strategies, highlighting that pre-training models with WSO enhances their adaptability for downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。