用随机微分方程揭示自适应优化器中噪声的作用机制
Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
- 构建了SignSGD、AdamW、RMSpropW的SDE模型，精确描述其动态行为
- 发现自适应方法在重尾噪声下比SGD更鲁棒，收敛速度更快
- 适合研究优化器理论或训练调参的科研人员参考
尽管自适应优化方法在深度学习中表现优异，但其理论理解仍不充分。本文为SignSGD、RMSprop(W)和Adam(W)构建了新的随机微分方程（SDE）模型，能定量刻画这些优化器的行为，并揭示自适应性、梯度噪声与曲率之间的复杂关系。对SignSGD的新分析显示，其在收敛速度、平稳分布及对重尾噪声的鲁棒性方面显著优于SGD。扩展至AdamW和RMSpropW后发现噪声作用更为复杂。实验通过Euler-Maruyama离散化在MLP、CNN、ResNet和Transformer上验证了模型准确性，优于先前针对Adam和RMSprop的SDE。本研究为最佳训练实践和新型缩放规则提供了理论洞见。
原文摘要 · Abstract (English)
Despite the vast empirical evidence supporting the efficacy of adaptive optimization methods in deep learning, their theoretical understanding is far from complete. This work introduces novel SDEs for commonly used adaptive optimizers: SignSGD, RMSprop(W), and Adam(W). These SDEs offer a quantitatively accurate description of these optimizers and help illuminate an intricate relationship between adaptivity, gradient noise, and curvature. Our novel analysis of SignSGD highlights a noteworthy and precise contrast to SGD in terms of convergence speed, stationary distribution, and robustness to heavy-tail noise. We extend this analysis to AdamW and RMSpropW, for which we observe that the role of noise is much more complex. Crucially, we support our theoretical analysis with experimental evidence by verifying our insights: this includes numerically integrating our SDEs using Euler-Maruyama discretization on various neural network architectures such as MLPs, CNNs, ResNets, and Transformers. Our SDEs accurately track the behavior of the respective optimizers, especially when compared to previous SDEs derived for Adam and RMSprop. We believe our approach can provide valuable insights into best training practices and novel scaling rules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。