揭示Adam比SGD快的关键在于二阶矩归一化，理论首次证明其收敛性更优。
Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails
- 通过停时与鞅分析，发现Adam的二阶矩归一化机制是性能优势来源。
- 理论上证明Adam在高概率下收敛依赖δ⁻¹/²，而SGD至少需δ⁻¹。
- 为理解Adam高效提供新视角，适合研究优化算法的学者参考。
尽管在众多应用中Adam表现出比SGD更快的实证收敛速度，但现有理论给出的保证与SGD相当，难以解释这一性能差距。本文揭示了Adam中的关键二阶矩归一化机制，并基于经典有界方差模型（即二阶矩假设）发展出停时与鞅分析方法，首次严格区分了Adam与SGD的高概率收敛行为。具体而言，我们证明：Adam的收敛性对置信参数δ的依赖为δ⁻¹/²，而SGD对应的高概率保证必然至少达到δ⁻¹，实现了理论上的分离。
原文摘要 · Abstract (English)
Despite Adam demonstrating faster empirical convergence than SGD in many applications, much of the existing theory yields guarantees essentially comparable to those of SGD, leaving the empirical performance gap insufficiently explained. In this paper, we uncover a key second-moment normalization in Adam and develop a stopping-time/martingale analysis that provably distinguishes Adam from SGD under the classical bounded variance model (a second moment assumption). In particular, we establish the first theoretical separation between the high-probability convergence behaviors of the two methods: Adam achieves a $δ^{-1/2}$ dependence on the confidence parameter $δ$, whereas corresponding high-probability guarantee for SGD necessarily incurs at least a $δ^{-1}$ dependence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。