揭示AdamW优化器在高维场景下的理论收敛速度，为实际训练提供理论依据。
On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm
- 基于ℓ₁范数分析AdamW的收敛性，给出与梯度维度相关的理论速率
- 实证表明梯度ℓ₁与ℓ₂范数呈√d量级关系，支持理论结论
- 结果适用于主流优化器变体NAdamW，对大模型训练有指导意义
作为训练大语言模型的默认优化器，AdamW在深度学习中表现卓越，但其收敛行为缺乏充分的理论理解。本文建立了以ℓ₁范数衡量的收敛速率：\frac{1}{K}\sum_{k=1}^K E[||\nabla f(x^k)||_1] \leq O(\frac{\sqrt{d}C}{K^{1/4}})，其中K为迭代次数，d为模型维度，C与SGD最优收敛率中的常数一致。理论上，对任意高维向量有||\nabla f(x)||_2 \ll ||\nabla f(x)||_1 \leq \sqrt{d}||\nabla f(x)||_2；当梯度各分量服从\mathcal N(0,1)时，E[||\nabla f(x)||_1] \geq \sqrt{\frac{2d}{\pi}}E[||\nabla f(x)||_2]。实验证明，在真实深度学习任务中，||\nabla f(x)||_1 = \varTheta(\sqrt{d})||\nabla f(x)||_2，二者量级一致。这表明该收敛率在理想情况下等价于SGD的最优\frac{1}{K}\sum_{k=1}^K E[||\nabla f(x^k)||_2] \leq O(\frac{C}{K^{1/4}})。研究还扩展至采用双动量机制的NAdamW，证明其保持相同收敛速率。
原文摘要 · Abstract (English)
As the default optimizer for training large language models, AdamW has achieved remarkable success in deep learning. However, its convergence behavior is not theoretically well-understood. This paper establishes the convergence rate $\frac{1}{K}\sum_{k=1}^KE\left[||\nabla f(x^k)||_1\right]\leq O(\frac{\sqrt{d}C}{K^{1/4}})$ for AdamW measured by $\ell_1$ norm, where $K$ represents the iteration number, $d$ denotes the model dimension, and $C$ matches the constant in the optimal convergence rate of SGD. Theoretically, we have $||\nabla f(x)||_2\ll ||\nabla f(x)||_1\leq \sqrt{d}||\nabla f(x)||_2$ for any high-dimensional vector $x$ and $E\left[||\nabla f(x)||_1\right]\geq\sqrt{\frac{2d}π}E\left[||\nabla f(x)||_2\right]$ when each element of $\nabla f(x)$ is generated from Gaussian distribution $\mathcal N(0,1)$. Empirically, our experimental results on real-world deep learning tasks reveal $||\nabla f(x)||_1=\varTheta(\sqrt{d})||\nabla f(x)||_2$. Both support that our convergence rate can be considered to be analogous to the optimal $\frac{1}{K}\sum_{k=1}^KE\left[||\nabla f(x^k)||_2\right]\leq O(\frac{C}{K^{1/4}})$ convergence rate of SGD in the ideal case. We also extend our result to NAdamW, an AdamW variant that employs a double-momentum mechanism, and demonstrate that it maintains the same convergence rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。