证明Adam优化器收敛速度比RMSprop更快，达到理论最优阶。
Sharp higher order convergence rates for the Adam optimizer
- 通过分析Hessian条件数，揭示Adam的收敛速率优于标准梯度下降
- 在严格局部极小点附近，Adam收敛率可达(√x−1)/(√x+1)，优于RMSprop的(x−1)/(x+1)
- 为Adam高效性提供理论支撑，适合研究优化算法的学者参考
基于梯度下降的优化方法是训练深度神经网络的主流选择。除标准梯度下降外，常采用包含加速技术（如动量法）和自适应技术（如RMSprop）的改进方法。如今最流行的复杂优化方案当属2014年由Kingma和Ba提出的Adam优化器。研究此类方法的收敛速度是一个重要课题。1964年Polyak曾证明，标准梯度下降在严格局部极小点附近的收敛率为 (x - 1)(x + 1)^{-1}，而动量法可达到更优的 (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}，其中 x ∈ (1,∞) 为损失函数在极小点处Hessian矩阵的条件数（最大与最小特征值之比）。本文关键贡献在于证明：尽管未显式引入动量，但Adam仍能实现严格更快的收敛率 (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}；而仅含自适应的RMSprop则收敛于较慢的 (x - 1)(x + 1)^{-1}。
原文摘要 · Abstract (English)
Gradient descent based optimization methods are the methods of choice to train deep neural networks in machine learning. Beyond the standard gradient descent method, also suitable modified variants of standard gradient descent involving acceleration techniques such as the momentum method and/or adaptivity techniques such as the RMSprop method are frequently considered optimization methods. These days the most popular of such sophisticated optimization schemes is presumably the Adam optimizer that has been proposed in 2014 by Kingma and Ba. A highly relevant topic of research is to investigate the speed of convergence of such optimization methods. In particular, in 1964 Polyak showed that the standard gradient descent method converges in a neighborhood of a strict local minimizer with rate (x - 1)(x + 1)^{-1} while momentum achieves the (optimal) strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} where x \in (1,\infty) is the condition number (the ratio of the largest and the smallest eigenvalue) of the Hessian of the objective function at the local minimizer. It is the key contribution of this work to reveal that Adam also converges with the strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} while RMSprop only converges with the convergence rate (x - 1)(x + 1)^{-1}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。