发现Adam高效的关键在于动量参数相等的设定。
In Search of Adam's Secret Sauce
- 让Adam的动量参数beta1与beta2相等，简化优化器结构。
- 该设定在1500多个模型上表现稳定，优于其他简化版本。
- 揭示了Adam本质是梯度均值方差的在线估计算法。
理解Adam在训练基于Transformer的语言模型时的卓越效果，已成为优化领域的重要研究课题。为深入探究，已有研究提出如符号梯度和符号动量等简化版Adam。本文通过大规模实验——在不同数据配置和规模下训练超过1500个语言模型——比较Adam与其已知简化变体的表现。结果表明，尽管符号动量方法比SGD更快，但在经过对动量、裁剪设置和学习率的细致调优后，仍持续劣于Adam。我们的分析揭示了一个关键发现：将Adam的动量参数约束为相等（即 beta1 = beta2），既能保持近似最优性能，又支持新的理论重构。在此设定下，Adam可被解释为一种自然的在线算法，用于估计梯度的均值与方差，其理论基础源自均场高斯变分推断视角。
原文摘要 · Abstract (English)
Understanding the remarkable efficacy of Adam when training transformer-based language models has become a central research topic within the optimization community. To gain deeper insights, several simplifications of Adam have been proposed, such as the signed gradient and signed momentum methods. In this work, we conduct an extensive empirical study - training over 1500 language models across different data configurations and scales - comparing Adam to several known simplified variants. We find that signed momentum methods are faster than SGD, but consistently underperform relative to Adam, even after careful tuning of momentum, clipping setting and learning rates. However, our analysis reveals a compelling option that preserves near-optimal performance while allowing for new insightful reformulations: constraining the Adam momentum parameters to be equal, beta1 = beta2. Beyond robust performance, this choice affords new theoretical insights, highlights the "secret sauce" on top of signed momentum, and grants a precise statistical interpretation: we show that Adam in this setting implements a natural online algorithm for estimating the mean and variance of gradients-one that arises from a mean-field Gaussian variational inference perspective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。