自动调节学习率的梯度下降法，无需人工调参。
AutoGD: Automatic Learning Rate Selection for Gradient Descent
- 根据当前迭代自动判断增减学习率，无需预设调度策略。
- 在多种任务中表现优于传统方法，收敛速度接近最优理论值。
- 适合嵌入其他算法作为内循环，减少人工调参负担。
基于梯度的优化方法（如标准梯度下降）性能高度依赖学习率的选择。然而，确定合适的学习率调度通常需要大量人工调参。当此类方法作为其他算法的内循环时，要求用户调参可能不切实际。为此，我们提出 AutoGD：一种能自动判断每轮迭代是否应增大或减小学习率的梯度下降方法。我们证明了 AutoGD 的收敛性，并表明在无需平滑常数信息的情况下，可恢复广义函数类下梯度下降的最优收敛率（至多常数因子）。在多种经典优化问题及变分推断任务上的实验表明，该方法性能优异，并成功扩展至 AutoBFGS 与 AutoLBFGS。
原文摘要 · Abstract (English)
The performance of gradient-based optimization methods, such as standard gradient descent (GD), greatly depends on the choice of learning rate. However, it can require a non-trivial amount of user tuning effort to select an appropriate learning rate schedule. When such methods appear as inner loops of other algorithms, expecting the user to tune the learning rates may be impractical. To address this, we introduce AutoGD: a gradient descent method that automatically determines whether to increase or decrease the learning rate at a given iteration. We establish the convergence of AutoGD, and show that we can recover the optimal rate of GD (up to a constant) for a broad class of functions without knowledge of smoothness constants. Experiments on a variety of traditional problems and variational inference optimization tasks demonstrate strong performance of the method, along with its extensions to AutoBFGS and AutoLBFGS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。