提出非均匀光滑性下优化算法收敛新理论，解释为何Adam等更快。
Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
- 基于目标函数曲率与自身值呈仿射关系的假设，统一分析多种优化器。
- 证明在可分数据上，Logistic回归和软最大策略梯度的符号梯度下降线性收敛。
- 首次给出Adam/RMSProp比AdaGrad、GD等更快的理论下界，适合理论研究者。
近期工作分析了在非均匀光滑性假设下一阶优化方法的收敛性，该假设更贴近机器学习损失曲面的实际特征。本文将此假设推广至曲率是目标值仿射函数的优化问题，涵盖逻辑回归、广义线性模型（逻辑链接）、强化学习中的软最大策略梯度，以及一类神经网络。在此假设及梯度支配条件下，建立了梯度下降、RMSProp和Adam的确定性对角变体的通用收敛速率。结果表明，在可分数据上的逻辑回归和软最大策略梯度问题中，符号梯度下降具有线性收敛速度，且理论上优于标准梯度下降。此外，对于一类两层神经网络在可分数据上，使用固定步长和动量参数的RMSProp和Adam也能实现线性收敛。最后，我们给出了一个下界，证明在该假设下，RMSProp和Adam在理论上比AdaGrad、AMSGrad、梯度下降和重球动量方法更快。
原文摘要 · Abstract (English)
Recent work has analyzed the convergence of first-order methods under non-uniform smoothness assumptions that better model the loss landscape in machine learning tasks. We generalize this assumption to objectives whose curvature is an affine function of the objective value. This property is satisfied by a broad class of problems, including logistic regression, generalized linear models with a logistic link function, softmax policy gradient in reinforcement learning, and a class of neural networks. Under this assumption and gradient domination conditions, we establish a general convergence rate for the steepest descent method, and deterministic, diagonal variants of RMSProp and Adam. Our results imply that for logistic regression on separable data and the softmax policy gradient objective, sign GD converges linearly and is provably faster than GD. Furthermore, we show that for a class of two-layer neural networks on separable data, RMSProp and Adam can converge at a linear rate with a constant step-size and momentum parameter. Finally, we present a lower bound demonstrating that, under our assumption, RMSProp and Adam are provably faster than AdaGrad, AMSGrad, gradient descent, and heavy-ball momentum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。