arXiv:2505.04599cs.LGmath.OC2025-05ICLR被引 8

揭示自适应优化算法在非凸随机优化中的复杂度下界,说明其性能受初始差距和光滑性参数显著制约。

Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness

  • 研究自适应算法在$(L_0, L_1)$-光滑条件下的收敛复杂度下界
  • 证明AdaGrad-Norm变体至少需$Ω(Δ^2 L_1^2 σ^2 ε^{-4})$次梯度查询
  • 表明部分自适应算法在非凸问题中比SGD更差,适合关注理论边界的研究者

近期研究表明,流行自适应算法(如AdaGrad)在$(L_0, L_1)$-光滑条件下可实现收敛,但收敛速率对问题参数(如光滑常数)呈高阶多项式依赖。此类算法保证找到$ε$-驻点的复杂度可能远高于标准$L$-光滑设置下SGD达到的最优复杂度$Θ(ΔL σ^2 ε^{-4})$,其中$Δ$为初始最优性差距,$σ^2$为随机梯度方差。目前尚不清楚这种高阶依赖能否收紧。本文研究多种自适应优化算法在$(L_0, L_1)$-光滑设定下的复杂度下界,重点分析$Δ, L_0, L_1$的依赖关系。我们为三种AdaGrad变体提供了下界,表明至少存在$Δ, L_0, L_1$的二次依赖。特别地,我们证明去相关版AdaGrad-Norm至少需要$Ω(Δ^2 L_1^2 σ^2 ε^{-4})$次随机梯度查询才能找到$ε$-驻点。此外,我们也给出了具有广泛自适应步长的SGD的下界。结果表明,对于某些自适应算法,$(L_0, L_1)$-光滑设定在初始最优性差距与光滑常数方面本质上比标准光滑设定更困难。

原文摘要 · Abstract (English)

Recent results in non-convex stochastic optimization demonstrate the convergence of popular adaptive algorithms (e.g., AdaGrad) under the $(L_0, L_1)$-smoothness condition, but the rate of convergence is a higher-order polynomial in terms of problem parameters like the smoothness constants. The complexity guaranteed by such algorithms to find an $ε$-stationary point may be significantly larger than the optimal complexity of $Θ\left( ΔL σ^2 ε^{-4} \right)$ achieved by SGD in the $L$-smooth setting, where $Δ$ is the initial optimality gap, $σ^2$ is the variance of stochastic gradient. However, it is currently not known whether these higher-order dependencies can be tightened. To answer this question, we investigate complexity lower bounds for several adaptive optimization algorithms in the $(L_0, L_1)$-smooth setting, with a focus on the dependence in terms of problem parameters $Δ, L_0, L_1$. We provide complexity bounds for three variations of AdaGrad, which show at least a quadratic dependence on problem parameters $Δ, L_0, L_1$. Notably, we show that the decorrelated variant of AdaGrad-Norm requires at least $Ω\left( Δ^2 L_1^2 σ^2 ε^{-4} \right)$ stochastic gradient queries to find an $ε$-stationary point. We also provide a lower bound for SGD with a broad class of adaptive stepsizes. Our results show that, for certain adaptive algorithms, the $(L_0, L_1)$-smooth setting is fundamentally more difficult than the standard smooth setting, in terms of the initial optimality gap and the smoothness constants.

优化算法复杂度下界非凸优化自适应梯度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。