解释了学习率预热为何有效，从理论出发找到优化初期的曲率规律。
Why Do We Need Warm-up? A Theoretical Perspective
- 用改进的光滑性条件建模训练初期的梯度曲率变化。
- 证明自适应学习率比固定学习率收敛更快，且与线性预热效果相当。
- 适合研究优化机制或想理解预热原理的机器学习从业者。
学习率预热——在训练初期逐步提升学习率——已成为现代深度学习中的常见技巧，但其理论基础仍不清晰。本文提出一种基于 $(L_0, L_1)$-光滑性条件的推广模型，该条件以损失次优性为线性函数约束局部曲率，并具备良好的闭包性质。我们理论上和实验上均证明，该条件适用于常见神经网络架构，能准确刻画训练早期的优化景观曲率。根据此曲率条件自适应调整学习率，自然产生类似预热的调度策略。理论分析表明，该策略比固定学习率具有更优的收敛保证。在语言与视觉模型上的实验显示，仅一个参数的预热方案可媲美人工调优的线性预热，且优于无预热情况。
原文摘要 · Abstract (English)
Learning rate warm-up -- increasing the learning rate at the beginning of training -- has become a ubiquitous heuristic in modern deep learning, yet its theoretical foundations remain poorly understood. In this work, we provide a principled explanation for why warm-up improves training. We rely on a generalization of the $(L_0, L_1)$-smoothness condition, which bounds local curvature as a linear function of the loss suboptimality and exhibits desirable closure properties. We show -- both theoretically and empirically -- that this condition is satisfied by common neural architectures and accurately captures the curvature of the optimization landscape early in training. Adapting the learning rate in response to this curvature condition naturally induces a warm-up-like schedule, and we show that this choice yields provably faster convergence guarantees than using a fixed learning rate. Experiments on language and vision models show that the resulting one-parameter warm-up schedule can match tuned linear warm-up and improve over no warm-up.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。