提出新算法实现广义光滑函数优化的近最优收敛速度。
Near-Optimal Convergence of Accelerated Gradient Methods under Generalized and $(L_0, L_1)$-Smoothness
- 设计新李雅普诺夫函数与算法框架,突破原有加速梯度方法瓶颈。
- 在小误差下达到最优复杂度 $O(\sqrt{\ell(0)} R / \sqrt{\varepsilon})$,无额外因子。
- 适用于 $(L_0, L_1)$-光滑等情形,对研究优化算法者极具参考价值。
我们研究满足新型 $\ell$-光滑性条件的凸优化问题的一阶方法,该条件推广了 $L$-光滑性和 $(L_0, L_1)$-光滑性。尽管在 $L$-光滑性下加速梯度下降(AGD)可达到最优复杂度 $O(\sqrt{L} R / \sqrt{\varepsilon})$,但现有扩展在 $\ell$-光滑性下或依赖初始梯度、或出现 $L_1 R$ 的指数因子、或需昂贵辅助子程序,无法实现 $O(\sqrt{\ell(0)} R / \sqrt{\varepsilon})$ 的速率。本文通过引入新李雅普诺夫函数并设计新算法,首次在小误差 $\varepsilon$ 下实现 $O(\sqrt{\ell(0)} R / \sqrt{\varepsilon})$ 的预言机复杂度,且适用于几乎任意 $\ell$。例如,在 $(L_0, L_1)$-光滑性下,其界 $O(\sqrt{L_0} R / \sqrt{\varepsilon})$ 在小 $\varepsilon$ 下被证明为最优,完全消除了先前加速算法中的非常数乘性因子。
原文摘要 · Abstract (English)
We study first-order methods for convex optimization problems with functions $f$ satisfying the recently proposed $\ell$-smoothness condition $||\nabla^{2}f(x)|| \le \ell\left(||\nabla f(x)||\right),$ which generalizes the $L$-smoothness and $(L_{0},L_{1})$-smoothness. While accelerated gradient descent AGD is known to reach the optimal complexity $O(\sqrt{L} R / \sqrt{\varepsilon})$ under $L$-smoothness, where $\varepsilon$ is an error tolerance and $R$ is the distance between a starting and an optimal point, existing extensions to $\ell$-smoothness either incur extra dependence on the initial gradient, suffer exponential factors in $L_{1} R$, or require costly auxiliary sub-routines, leaving open whether an AGD-type $O(\sqrt{\ell(0)} R / \sqrt{\varepsilon})$ rate is possible for small-$\varepsilon$, even in the $(L_{0},L_{1})$-smoothness case. We resolve this open question. Leveraging a new Lyapunov function and designing new algorithms, we achieve $O(\sqrt{\ell(0)} R / \sqrt{\varepsilon})$ oracle complexity for small-$\varepsilon$ and virtually any $\ell$. For instance, for $(L_{0},L_{1})$-smoothness, our bound $O(\sqrt{L_0} R / \sqrt{\varepsilon})$ is provably optimal in the small-$\varepsilon$ regime and removes all non-constant multiplicative factors present in prior accelerated algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。