通过搜索找到神经网络训练中接近最优的学习率调度形状。
What do near-optimal learning rate schedules look like?
- 设计搜索方法分离学习率形状与基础值，避免比较偏差。
- 在三种任务上发现最优调度均含预热和衰减阶段。
- 权重衰减会显著影响最优调度形状，适合调参研究者参考。
神经网络训练中的一个基本未解问题：针对特定任务，最佳学习率调度形状是什么？学习率调度是训练成败的关键因素，但除了预热和衰减外，尚无共识。为此，我们设计了一种搜索程序，在参数化调度族中寻找最优形状。该方法将调度形状与基础学习率分离，避免后者主导跨调度比较。我们在三种任务上应用该方法：线性回归、CIFAR-10图像分类、Wikitext103小规模语言建模。结果表明，搜索过程确实找到了近似最优的调度。发现预热和衰减是良好调度的稳健特征；常见调度族在这些任务上并非最优。进一步分析发现，权重衰减对最优调度形状有显著影响。据我们所知，这是迄今对深度神经网络训练中近似最优调度形状最全面的研究。
原文摘要 · Abstract (English)
A basic unanswered question in neural network training is: what is the best learning rate schedule shape for a given workload? The choice of learning rate schedule is a key factor in the success or failure of the training process, but beyond having some kind of warmup and decay, there is no consensus on what makes a good schedule shape. To answer this question, we designed a search procedure to find the best shapes within a parameterized schedule family. Our approach factors out the schedule shape from the base learning rate, which otherwise would dominate cross-schedule comparisons. We applied our search procedure to a variety of schedule families on three workloads: linear regression, image classification on CIFAR-10, and small-scale language modeling on Wikitext103. We showed that our search procedure indeed generally found near-optimal schedules. We found that warmup and decay are robust features of good schedules, and that commonly used schedule families are not optimal on these workloads. Finally, we explored how the outputs of our shape search depend on other optimization hyperparameters, and found that weight decay can have a strong effect on the optimal schedule shape. To the best of our knowledge, our results represent the most comprehensive results on near-optimal schedule shapes for deep neural network training, to date.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。