随机遮蔽参数更新可显著提升大模型训练效果
On Surprising Effectiveness of Masking Updates in Adaptive Optimizers
- 用随机遮蔽替代部分参数更新，实现更平滑优化轨迹
- 新方法Magma在10亿参数模型上降低困惑度超19%
- 无需额外计算开销，可直接替换现有优化器
大语言模型训练几乎完全依赖密集自适应优化器及其复杂的预条件机制。我们提出，随机遮蔽参数更新同样高效，其变体RMSProp在多项实验中持续优于当前最先进的优化器。分析表明，随机遮蔽引入了基于曲率的几何正则化，使优化路径更加平滑。受此启发，我们提出动量-梯度对齐遮蔽（Magma），通过动量与梯度对齐调节遮蔽更新。大规模大模型预训练实验显示，Magma是自适应优化器的简单替代方案，性能稳定提升且计算开销可忽略。值得注意的是，在10亿参数模型上，相比Adam和Muon，Magma分别将困惑度降低超过19%和9%。
原文摘要 · Abstract (English)
Training large language models (LLMs) relies almost exclusively on dense adaptive optimizers with increasingly sophisticated preconditioners. We challenge this by showing that randomly masking parameter updates can be highly effective, with a masked variant of RMSProp consistently outperforming recent state-of-the-art optimizers. Our analysis reveals that the random masking induces a curvature-dependent geometric regularization that smooths the optimization trajectory. Motivated by this finding, we introduce Momentum-aligned gradient masking (Magma), which modulates the masked updates using momentum-gradient alignment. Extensive LLM pre-training experiments show that Magma is a simple drop-in replacement for adaptive optimizers with consistent gains and negligible computational overhead. Notably, for the 1B model size, Magma reduces perplexity by over 19\% and 9\% compared to Adam and Muon, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。