将Nesterov加速与自适应梯度法GRAAL结合,实现更快更智能的优化。
Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimization
- 用局部曲率自适应调整步长,无需调参或线搜索
- 在L-光滑和更一般条件下逼近最优收敛速度
- 适合追求高效收敛的机器学习优化场景
本文研究连续可微凸函数的最小化问题。近期,Malitsky(2020)和Alacaoglu等(2023)提出了自适应一阶方法GRAAL,通过估计目标函数的局部曲率自动确定步长,无需线搜索或超参数调优,并达到固定步长梯度下降的标准迭代复杂度:$\mathcal{O}(L\lVert x_0-x^*\rVert^2/ε)$(对L-光滑函数)。但一个自然问题是:能否将GRAAL的收敛速度加速至与Nesterov(1983)加速梯度下降相同的最优复杂度 $\mathcal{O}(\sqrt{L\lVert x_0-x^*\rVert^2/ε})$?尽管已有工作尝试(Li & Lan, 2025;Suh & Ma, 2025),但现有加速算法对局部曲率的自适应能力仍有限。本文解决了这一问题,提出带Nesterov加速的GRAAL,其步长仍以几何或线性速率适应局部曲率,表现如非加速版GRAAL。我们证明该算法在L-光滑函数下以及更一般的$(L_0,L_1)$-光滑假设(Zhang et al., 2019)下均能达到近似最优的迭代复杂度。
原文摘要 · Abstract (English)
In this paper, we focus on the problem of minimizing a continuously differentiable convex objective function, $\min_x f(x)$. Recently, Malitsky (2020); Alacaoglu et al.(2023) developed an adaptive first-order method, GRAAL. This algorithm computes stepsizes by estimating the local curvature of the objective function without any line search procedures or hyperparameter tuning, and attains the standard iteration complexity $\mathcal{O}(L\lVert x_0-x^*\rVert^2/ε)$ of fixed-stepsize gradient descent for $L$-smooth functions. However, a natural question arises: is it possible to accelerate the convergence of GRAAL to match the optimal complexity $\mathcal{O}(\sqrt{L\lVert x_0-x^*\rVert^2/ε})$ of the accelerated gradient descent of Nesterov (1983)? Although some attempts have been made by Li and Lan (2025); Suh and Ma (2025), the ability of existing accelerated algorithms to adapt to the local curvature of the objective function is highly limited. We resolve this issue and develop GRAAL with Nesterov acceleration, which can adapt its stepsize to the local curvature at a geometric, or linear, rate just like non-accelerated GRAAL. We demonstrate the adaptive capabilities of our algorithm by proving that it achieves near-optimal iteration complexities for $L$-smooth functions, as well as under a more general $(L_0,L_1)$-smoothness assumption (Zhang et al., 2019).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。