提出自适应学习率调度，让大模型训练自动确定最佳预热时长。
Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers
- 基于曲率随误差减小的假设，推导出自然出现的预热+衰减策略。
- 在LLaMA模型上验证，自适应预热优于或等同于人工调参结果。
- 仅依赖标准超参数，无需额外搜索，适合大模型高效训练。
我们研究了范数约束优化器（如 Muon 和 Lion）的自适应学习率调度。提出一种广义平滑性假设，即局部曲率随次优性差距减小，且在优化轨迹上实证成立。在此假设下，证明了恰当学习率选择可保证收敛，其中预热后衰减的模式自然产生，而非人为强加。基于此理论，我们设计了一种仅依赖标准超参数的实用学习率调度器，能自动调整训练初期的预热时长。在使用 LLaMA 架构的大语言模型预训练中评估，该方法在所有设置下均表现优于或等同于最优手动调参的预热方案，且无需额外超参数搜索。代码已开源。
原文摘要 · Abstract (English)
We study adaptive learning rate scheduling for norm-constrained optimizers (e.g., Muon and Lion). We introduce a generalized smoothness assumption under which local curvature decreases with the suboptimality gap and empirically verify that this behavior holds along optimization trajectories. Under this assumption, we establish convergence guarantees under an appropriate choice of learning rate, for which warm-up followed by decay arises naturally from the proof rather than being imposed heuristically. Building on this theory, we develop a practical learning rate scheduler that relies only on standard hyperparameters and adapts the warm-up duration automatically at the beginning of training. We evaluate this method on large language model pretraining with LLaMA architectures and show that our adaptive warm-up selection consistently outperforms or at least matches the best manually tuned warm-up schedules across all considered setups, without additional hyperparameter search. Our source code is available at https://github.com/brain-lab-research/llm-baselines/tree/warmup
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。