学习率衰减让调参更鲁棒，大幅降低搜索成本
Learning Rate Annealing Improves Tuning Robustness in Stochastic Optimization
- 用多项式衰减学习率，减少初始参数误差影响
- 收敛速度受参数误差影响仅呈亚线性增长
- 适合大规模模型训练，降低调参计算开销
随机梯度方法中的学习率是关键超参数，传统网格搜索调优成本高昂，尤其在训练含数十亿参数的现代模型时。本文理论分析表明，以多项式速率（如常用余弦调度）衰减学习率可显著提升对初始参数误设的鲁棒性。在随机凸优化设定下，使用退火策略的SGD收敛速率对乘性误设因子ρ（即网格分辨率）呈亚线性依赖，达到O(ρ^(1/(2p+1))/√T)，其中p为多项式衰减速率阶数，T为迭代步数。而固定步长或反平方根调度的收敛速率则为O(ρ/√T)，呈线性依赖。实验验证了退火策略相比固定步长具有更强鲁棒性，这对实际训练中降低超参数搜索计算开销具有重要意义。
原文摘要 · Abstract (English)
The learning rate in stochastic gradient methods is a critical hyperparameter that is notoriously costly to tune via standard grid search, especially for training modern large-scale models with billions of parameters. We identify a theoretical advantage of learning rate annealing schemes that decay the learning rate to zero at a polynomial rate, such as the widely-used cosine schedule, by demonstrating their increased robustness to initial parameter misspecification due to a coarse grid search. We present an analysis in a stochastic convex optimization setup demonstrating that the convergence rate of stochastic gradient descent with annealed schedules depends sublinearly on the multiplicative misspecification factor $ρ$ (i.e., the grid resolution), achieving a rate of $O(ρ^{1/(2p+1)}/\sqrt{T})$ where $p$ is the degree of polynomial decay and $T$ is the number of steps. This is in contrast to the $O(ρ/\sqrt{T})$ rate obtained under the inverse-square-root and fixed stepsize schedules, which depend linearly on $ρ$. Experiments confirm the increased robustness compared to tuning with a fixed stepsize, that has significant implications for the computational overhead of hyperparameter search in practical training scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。