Adam比SGD更优，源于对损失函数∞-几何的利用。
Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity
- 基于∞-范数光滑性重新分析Adam收敛性
- 实测显示换掉∞-几何后Adam性能暴跌
- 适合研究优化器理论与模型训练机制者
Adam在训练语言模型时表现优于SGD，但其优势缺乏理论解释。以往对Adam和SGD的收敛性分析主要关注迭代步数T，且在非凸情形下均已达到最小最大最优，均为~O(T^{-1/4})。本文认为，Adam的核心优势在于利用了损失函数的优良ℓ∞-几何结构。我们提出新的收敛分析框架，假设损失函数在ℓ∞-几何下平滑，而非常见的ℓ2-几何，该假设使GPT-2和ResNet模型的平滑常数显著降低。实验验证：当ℓ∞-几何被破坏时，Adam性能大幅下降，而SGD仍保持稳定。我们还将分析扩展至分块版Adam，引入新的分块平滑性假设。
原文摘要 · Abstract (English)
Adam outperforms SGD when training language models. Yet this advantage is not well-understood theoretically -- previous convergence analysis for Adam and SGD mainly focuses on the number of steps $T$ and is already minimax-optimal in non-convex cases, which are both $\widetilde{O}(T^{-1/4})$. In this work, we argue that the exploitation of nice $\ell_\infty$-geometry is the key advantage of Adam over SGD. More specifically, we give a new convergence analysis for Adam under novel assumptions that loss is smooth under $\ell_\infty$-geometry rather than the more common $\ell_2$-geometry, which yields a much better empirical smoothness constant for GPT-2 and ResNet models. Our experiments confirm that Adam performs much worse when the favorable $\ell_\infty$-geometry is changed while SGD provably remains unaffected. We also extend the convergence analysis to blockwise Adam under novel blockwise smoothness assumptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。