发现优化器新关联，改进AdEMAMix提升训练效率。
Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
- 将加速SGD理论与新型优化器统一为同一框架。
- AdEMAMix在语言建模任务中表现最优，超越其他方法。
- 提出简化版AdEMAMix，性能不变但参数更少。
深度学习优化新算法如Schedule-Free、AdEMAMix、MARS和Lion，均对传统动量机制进行了修改。另一研究方向通过解耦动量系数与梯度权重，在噪声主导场景下实现了随机梯度下降的理论加速。本文揭示了这两类工作间的明确联系。基于初步实验（150M语言建模任务），发现最接近加速版SGD的AdEMAMix表现最佳。据此，我们提出Simplified-AdEMAMix：在大、小批量设置下保持与AdEMAMix相当性能的同时，消除两个动量项的需求。代码已开源：https://github.com/DepenM/Simplified-AdEMAMix/
原文摘要 · Abstract (English)
Recent advancements in deep learning optimization have introduced new algorithms, such as Schedule-Free optimizers, AdEMAMix, MARS and Lion which modify traditional momentum mechanisms. In a separate line of work, theoretical acceleration of stochastic gradient descent (SGD) in noise-dominated regime has been achieved by decoupling the momentum coefficient from the current gradient's weight. In this paper, we establish explicit connections between these two lines of work. We substantiate our theoretical findings with preliminary experiments on a 150m language modeling task. We find that AdEMAMix, which most closely resembles accelerated versions of stochastic gradient descent, exhibits superior performance. Building on these insights, we introduce a modification to AdEMAMix, termed Simplified-AdEMAMix, which maintains the same performance as AdEMAMix across both large and small batch-size settings while eliminating the need for two different momentum terms. The code for Simplified-AdEMAMix is available on the repository: https://github.com/DepenM/Simplified-AdEMAMix/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。