从符号下降视角简化Adam收敛证明,首次实现最优收敛率。
Simple Convergence Proof of Adam From a Sign-like Descent Perspective
- 将Adam重释为符号类优化器,避免复杂假设
- 在弱条件下达成O(1/T^1/4)最优收敛率
- 揭示动量关键作用,提供调参实用建议
Adam是训练深度神经网络最有效的优化器之一,尽管其经验表现优异,但理论收敛分析仍不完善。现有工作多将其视为带动量的预条件随机梯度下降,需强假设与复杂技巧,导致证明冗长且难验证。本文提出新视角:将Adam视为符号类优化器,表达为 $\bm{x}_{t+1} = \bm{x}_t - γ_t \frac{|\bm{m}_t|}{\sqrt{\bm{v}_t}+ε} \circ \rm Sign(\bm{m}_t)$。该形式显著简化分析。首次在温和条件下证明,无需依赖模型维度或数值稳定性参数 $ε$,在广义 $p$-仿射方差和 $(L_0, L_1, q)$-光滑性假设下,可达到最优收敛率 ${\cal O}(\frac{1}{T^{\sfrac{1}{4}}})$,优于以往 ${\cal O}(\frac{\ln T}{T^{\sfrac{1}{4}}})$。理论还揭示动量对收敛的关键作用,并提供学习率调优指导,弥合理论与实践差距。
原文摘要 · Abstract (English)
Adam is widely recognized as one of the most effective optimizers for training deep neural networks (DNNs). Despite its remarkable empirical success, its theoretical convergence analysis remains unsatisfactory. Existing works predominantly interpret Adam as a preconditioned stochastic gradient descent with momentum (SGDM), formulated as $\bm{x}_{t+1} = \bm{x}_t - \frac{γ_t}{{\sqrt{\bm{v}_t}+ε}} \circ \bm{m}_t$. This perspective necessitates strong assumptions and intricate techniques, resulting in lengthy and opaque convergence proofs that are difficult to verify and extend. In contrast, we propose a novel interpretation by treating Adam as a sign-like optimizer, expressed as $\bm{x}_{t+1} = \bm{x}_t - γ_t \frac{|\bm{m}_t|}{{\sqrt{\bm{v}_t}+ε}} \circ {\rm Sign}(\bm{m}_t)$. This reformulation significantly simplifies the convergence analysis. For the first time, with some mild conditions, we prove that Adam achieves the optimal rate of ${\cal O}(\frac{1}{T^{\sfrac{1}{4}}})$ rather than the previous ${\cal O} \left(\frac{\ln T}{T^{\sfrac{1}{4}}}\right)$ under weak assumptions of the generalized $p$-affine variance and $(L_0, L_1, q)$-smoothness, without dependence on the model dimensionality or the numerical stability parameter $ε$. Additionally, our theoretical analysis provides new insights into the role of momentum as a key factor ensuring convergence and offers practical guidelines for tuning learning rates in Adam, further bridging the gap between theory and practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。