调好参数后，SGD能追上Adam在小批量语言模型中的表现。
Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
- 系统对比动量版SGD与Adam，控制变量研究性能差异
- 小批量下经充分调参的SGD可达到与Adam相当效果
- 揭示批量大小对训练动态的影响机制，适合优化研究者
Adam在语言模型中通常显著优于随机梯度下降（SGD），对此已有多种解释。本文通过全面调优的Transformer语言建模基准实验，系统研究了动量、梯度裁剪和批量大小对SGD与Adam性能差距的影响。实证发现：若调参得当，带动量的SGD在小批量设置下可与Adam表现相当。我们重新审视了现有解释——如重尾类别失衡、方向性尖锐性、海森异质性——发现它们难以直接解释该现象。为弥合理解鸿沟，基于文献启发的简单二次模型及实际Transformer训练分析，我们借助随机微分方程模型，揭示了批量大小对训练动态的关键作用。
原文摘要 · Abstract (English)
Adam is known to perform significantly better than Stochastic Gradient Descent (SGD) in language models, a phenomenon for which a number of explanations have been proposed. In this work, we revisit this "optimizer gap" through a series of comprehensively tuned baseline training runs for language modeling with Transformers. We exhaustively study how momentum, gradient clipping, and batch size affect the gap between SGD and Adam. Our empirical findings show that SGD with momentum can actually perform similarly to Adam in small-batch settings, if tuned correctly. We revisit existing explanations for Adam's advantage, including heavy-tailed class imbalance, directional sharpness, and Hessian heterogeneity, which struggle to directly explain this phenomenon. Towards bridging this gap in our understanding, by analyzing our Transformer training runs and simple quadratic settings inspired by the literature, we provide new insights, driven by stochastic differential equation models, into the role of batch size on the training dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。