用动态系统建模训练过程，自动生成更优学习率调度
Optimization as a Dynamical System: Generative Schedules from Latent ODEs
- 将训练过程视为动力系统，从历史数据中学习潜在表征
- 在图像分类与文本预测任务上超越所有基线方法
- 无需修改优化器，可直接接入实验追踪平台
我们提出一种新的元学习方法，用于确定梯度下降的最优学习率调度。该方法利用超参数搜索中的训练记录，学习训练过程的潜在表示，并将其建模为动力系统。基于当前训练指标，预测未来能带来最佳长期验证性能的学习率调度。该调度器能泛化到未见过的训练动态，生成显著区别于现有参数化函数的专用调度方案。在使用CNN和ResNet进行图像分类，以及使用Transformer进行下一段预测的任务中，其表现全面优于所有对比基线。经此方法训练的模型位于损失曲面的平坦区域，因此具备更强泛化能力。该方法计算高效、与优化器无关，可轻松集成至机器学习实验追踪平台，实现神经网络从头训练的自动化。
原文摘要 · Abstract (English)
We present a new meta-learning method to determine the optimal learning rate schedule for gradient descent. It leverages training runs from a hyperparameter search to learn a latent representation of the training process, which is modeled as a dynamical system. Given current training metrics, it predicts the future learning rate schedule with the best long-term validation performance. Our scheduler generalizes beyond previously observed training dynamics and creates specialized schedules that deviate noticeably from even the best-performing parametric functions. It outperforms all baselines we compare to on results for image classification with CNN and ResNet models as well as for next-token prediction with a transformer model. The trained models are located in flatter regions of the loss landscape and thus provide better generalization than those trained with other schedules. Our method is computationally efficient, optimizer-agnostic, and can easily be layered on top of ML experiment-tracking platforms to streamline training of neural networks from scratch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。