用数学证明了老方法能自适应调学习率，并实现超线性收敛。
Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent
- 基于在线学习框架，首次严格分析了超梯度下降法的收敛性。
- 新方法自动找最优学习率，在凸问题上性能优于主流自适应算法。
- 适合追求高效、低内存消耗优化器的研究者或工程师使用。
本文研究了超梯度下降法（HDM）的收敛性质，该方法是25年前提出的用于随机一阶优化中自适应步长选择的启发式算法。我们首次在[Gao24]的在线学习框架下对其进行了严格的收敛性分析，并据此开发出具有理论与实证支持的新一代自适应梯度方法。值得注意的是，HDM能自动识别局部优化景观下的最优步长，实现局部超线性收敛。我们的分析解释了文献中报道的HDM不稳定性问题，并提出有效应对策略。此外，我们还提出了带有Heavy-ball和Nesterov动量的HDM变体。在确定性凸问题上的实验表明，带Heavy-ball动量的HDM（HDM-HB）表现出稳健性能，显著优于其他自适应一阶方法。更重要的是，HDM-HB通常能媲美高效实用的拟牛顿方法L-BFGS，同时使用更少内存且迭代成本更低。
原文摘要 · Abstract (English)
This paper investigates the convergence properties of the hypergradient descent method (HDM), a 25-year-old heuristic originally proposed for adaptive stepsize selection in stochastic first-order methods. We provide the first rigorous convergence analysis of HDM using the online learning framework of [Gao24] and apply this analysis to develop new state-of-the-art adaptive gradient methods with empirical and theoretical support. Notably, HDM automatically identifies the optimal stepsize for the local optimization landscape and achieves local superlinear convergence. Our analysis explains the instability of HDM reported in the literature and proposes efficient strategies to address it. We also develop two HDM variants with heavy-ball and Nesterov momentum. Experiments on deterministic convex problems show HDM with heavy-ball momentum (HDM-HB) exhibits robust performance and significantly outperforms other adaptive first-order methods. Moreover, HDM-HB often matches the performance of L-BFGS, an efficient and practical quasi-Newton method, using less memory and cheaper iterations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。