arXiv:2502.03701math.OCcs.LG2025-02被引 2

用海森矩阵信息自适应调整梯度步长,让优化更稳定高效。

First-ish Order Methods: Hessian-aware Scalings of Gradient Descent

  • 基于海森矩阵修正梯度方向,实现局部单位步长
  • 非凸环境下仍保证全局收敛,且在极小值点线性收敛
  • 对海森矩阵误差不敏感,适合实际大规模优化任务

梯度下降是机器学习中大规模优化的核心方法,但其性能高度依赖学习率选择。传统方法缺乏自然缩放机制,常需代价高昂的线搜索或启发式调参。本文提出一种基于海森矩阵的自适应缩放方法,通过考虑函数沿梯度方向的曲率,确保即使在非凸情况下也具有局部单位步长性质。当靠近满足二阶充分条件的局部极小值时,该方法可实现单位步长下的线性收敛。理论证明其在比标准Lipschitz梯度光滑性假设弱得多的条件下仍能全局收敛。即使海森矩阵信息不精确,在温和条件下局部单位步长和全局收敛性依然成立。我们在多种凸与非凸机器学习任务上验证了理论结果,展示了该方法的有效性。

原文摘要 · Abstract (English)

Gradient descent is the primary workhorse for optimizing large-scale problems in machine learning. However, its performance is highly sensitive to the choice of the learning rate. A key limitation of gradient descent is its lack of natural scaling, which often necessitates expensive line searches or heuristic tuning to determine an appropriate step size. In this paper, we address this limitation by incorporating Hessian information to scale the gradient direction. By accounting for the curvature of the function along the gradient, our adaptive, Hessian-aware scaling method ensures a local unit step size guarantee, even in nonconvex settings. Near a local minimum that satisfies the second-order sufficient conditions, our approach achieves linear convergence with a unit step size. We show that our method converges globally under a significantly weaker version of the standard Lipschitz gradient smoothness assumption. Even when Hessian information is inexact, the local unit step size guarantee and global convergence properties remain valid under mild conditions. Finally, we validate our theoretical results empirically on a range of convex and nonconvex machine learning tasks, showcasing the effectiveness of the approach.

优化算法梯度下降海森矩阵收敛性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。