用李普希茨带宽算法自动调学习率，更省时间、更省算力。
Learning Rate Optimization for Deep Neural Networks Using Lipschitz Bandits
- 基于李普希茨带宽理论设计自适应学习率搜索策略
- 比现有方法少用评估次数和每轮训练的迭代次数
- 适合追求高效训练的深度学习研究者和工程师
学习率是神经网络训练中的关键参数。合理调优的学习率能加速训练并提升测试准确率。本文提出一种基于李普希茨带宽的算法来优化神经网络的学习率。该方法与广泛使用的HyperOpt以及近期提出的带宽算法BLiE进行了对比。在多个神经网络架构上的实验表明，相比HyperOpt和BLiE，本方法能在更少的评估次数和更少的每轮训练迭代次数下找到更优的学习率。因此，该方法可实现更高效的神经网络训练，显著降低训练时间和计算成本。
原文摘要 · Abstract (English)
Learning rate is a crucial parameter in training of neural networks. A properly tuned learning rate leads to faster training and higher test accuracy. In this paper, we propose a Lipschitz bandit-driven approach for tuning the learning rate of neural networks. The proposed approach is compared with the popular HyperOpt technique used extensively for hyperparameter optimization and the recently developed bandit-based algorithm BLiE. The results for multiple neural network architectures indicate that our method finds a better learning rate using a) fewer evaluations and b) lesser number of epochs per evaluation, when compared to both HyperOpt and BLiE. Thus, the proposed approach enables more efficient training of neural networks, leading to lower training time and lesser computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。