对比多种学习率控制方法，发现现有方案在复杂任务中表现不稳定，亟需算法选择机制。
Revisiting Learning Rate Control
- 对比多保真超参优化、固定调度与无超参学习等方法
- 多数方法在特定任务表现好，但跨场景可靠性差
- 建议引入可微调与元学习新方向，提升适配性
学习率是深度学习中最重要的超参数之一，其控制方法一直是AutoML和深度学习研究的热点。从经典优化到基于梯度统计的在线调度，方法多样。本文对比不同范式，评估当前学习率控制的现状。发现多保真超参数优化、固定超参数调度以及无超参数学习方法在特定深度学习任务上表现优异，但在不同设置下缺乏稳定性。这凸显了算法选择方法在学习率控制中的缺失，而该问题至今未被AutoML和深度学习社区重视。同时观察到，随着模型与任务复杂度增加，即使结合多保真策略，超参数优化方法的有效性也在下降。未来应聚焦更相关的测试任务，并探索可微调方法与元学习等新方向，以显著增强AutoML对这一关键因素的影响。
原文摘要 · Abstract (English)
The learning rate is one of the most important hyperparameters in deep learning, and how to control it is an active area within both AutoML and deep learning research. Approaches for learning rate control span from classic optimization to online scheduling based on gradient statistics. This paper compares paradigms to assess the current state of learning rate control. We find that methods from multi-fidelity hyperparameter optimization, fixed-hyperparameter schedules, and hyperparameter-free learning often perform very well on selected deep learning tasks but are not reliable across settings. This highlights the need for algorithm selection methods in learning rate control, which have been neglected so far by both the AutoML and deep learning communities. We also observe a trend of hyperparameter optimization approaches becoming less effective as models and tasks grow in complexity, even when combined with multi-fidelity approaches for more expensive model trainings. A focus on more relevant test tasks and new promising directions like finetunable methods and meta-learning will enable the AutoML community to significantly strengthen its impact on this crucial factor in deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。