arXiv:2506.12648math.OCcs.LG2025-06被引 5

提出新平滑性定义,让自适应步长在理论上也更优

Glocal Smoothness: Line search and adaptive step sizes can help in theory too!

  • 用函数本身的性质定义全局与局部平滑性,摆脱迭代依赖
  • 证明线搜索比固定步长收敛更快,某些场景下优于加速方法
  • 适用于梯度下降、随机优化等多类算法,理论分析更统一

一阶优化算法的迭代复杂度通常基于梯度的全局Lipschitz常数,使用固定步长可达到近似最优。然而实际中许多目标函数存在局部Lipschitz常数较小的区域,允许使用更大步长。已有研究提出多种局部Lipschitz假设,表明自适应步长或线搜索能提升收敛速度。但这些结果多依赖于算法迭代轨迹,难以跨方法比较复杂度。本文提出一种仅依赖函数性质的全局与局部(glocal)平滑性刻画,使得迭代复杂度上界可表示为与迭代无关的常数,从而实现不同算法间的公平比较。在此假设下,可直接证明线搜索优于固定步长,并在某些情形下,带线搜索的梯度下降复杂度优于固定步长的加速方法。进一步证明,glocal平滑性可提升Polyak步长、AdGD步长,以及坐标优化、随机梯度法、加速梯度法和非线性共轭梯度法的复杂度。

原文摘要 · Abstract (English)

Iteration complexities for optimizing smooth functions with first-order algorithms are typically stated in terms of a global Lipschitz constant of the gradient, and near-optimal results are then achieved using fixed step sizes. But many objective functions that arise in practice have regions with small Lipschitz constants where larger step sizes can be used. Many local Lipschitz assumptions have been proposed, which have led to results showing that adaptive step sizes and/or line searches yield improved convergence rates over fixed step sizes. However, these faster rates tend to depend on the iterates of the algorithm, which makes it difficult to compare the iteration complexities of different methods. We consider a simple characterization of global and local ("glocal") smoothness that only depends on properties of the function. This allows upper bounds on iteration complexities in terms of iterate-independent constants and enables us to compare iteration complexities between algorithms. Under this assumption it is straightforward to show the advantages of line searches over fixed step sizes and that, in some settings, gradient descent with line search has a better iteration complexity than accelerated methods with fixed step sizes. We further show that glocal smoothness can lead to improved complexities for the Polyak and AdGD step sizes, as well other algorithms including coordinate optimization, stochastic gradient methods, accelerated gradient methods, and non-linear conjugate gradient methods.

优化理论自适应步长收敛分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。