揭示自适应SGD在小学习率下的连续动态规律
Effective continuous equations for adaptive SGD: a stochastic analysis view
- 用随机修正方程框架分析自适应SGD的演化机制
- 发现采样噪声在极限下表现为独立布朗运动驱动参数与动量变化
- 给出学习率与超参数的缩放关系，适用于多种自适应方法
我们在小学习率条件下，对一些流行的自适应随机梯度下降（SGD）方法进行了理论分析。基于Li等人提出的随机修正方程框架，我们推导出这些方法的有效连续随机动力学。核心贡献在于：SGD中的采样诱导噪声在极限下表现为独立布朗运动，分别驱动参数和梯度二阶动量的演化。此外，借鉴Malladi等人的方法，我们研究了自适应方法中学习率与关键超参数之间的缩放关系，刻画了所有非平凡的极限动态行为。
原文摘要 · Abstract (English)
We present a theoretical analysis of some popular adaptive Stochastic Gradient Descent (SGD) methods in the small learning rate regime. Using the stochastic modified equations framework introduced by Li et al., we derive effective continuous stochastic dynamics for these methods. Our key contribution is that sampling-induced noise in SGD manifests in the limit as independent Brownian motions driving the parameter and gradient second momentum evolutions. Furthermore, extending the approach of Malladi et al., we investigate scaling rules between the learning rate and key hyperparameters in adaptive methods, characterising all non-trivial limiting dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。