MONA优化器让大模型训练更快更稳，能避开尖锐局部极小点。
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
- 在Muon基础上加入动量加速项，利用梯度差的指数移动平均计算
- 在10亿到680亿参数模型上均优于Muon和AdamW，最大模型用1万亿词训练
- 适合大规模语言模型训练，尤其在需要稳定收敛的场景下表现突出
Muon优化器为大语言模型训练提供了有前景的替代方案，通过矩阵正交化实现几何感知更新。然而，与所有一阶方法一样，Muon可能陷入尖锐局部极小点。本文提出MONA，将Muon的正交化框架与曲率感知加速相结合。MONA直接在梯度处理流程中引入加速项，该项基于梯度差的指数移动平均计算。我们提供了MONA的详细收敛分析，表明加速项可帮助逃离尖锐极小点，同时保留Muon的谱范数正则化特性。实验表明，MONA在三个尺度的Mixture-of-Experts预训练中（从10亿到680亿参数）均优于Muon和AdamW，最大模型在1万亿令牌上训练。此外，我们在MOE-68B-A3B模型上进行监督微调，并在通用能力、数学推理和代码生成基准上评估，MONA达到当前最优性能。
原文摘要 · Abstract (English)
The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Muon can become trapped in sharp local minima. In this work, we present MONA, an optimizer that bridges Muon's orthogonalization framework with curvature-aware acceleration. MONA adds an acceleration term directly into Muon's gradient processing pipeline. This term is calculated from the exponential moving average of gradient differences. We provide a detailed convergence analysis for MONA, showing that the acceleration term enables escape from sharp minima while preserving Muon's spectral-norm regularization. Empirically, MONA achieves better convergence and downstream task performance compared to both Muon and AdamW across three scales of Mixture-of-Experts pretraining, spanning from 1B to 68B parameters, with the largest model trained on 1 trillion tokens. Furthermore, we conduct supervised fine-tuning on the MOE-68B-A3B model and evaluate it on general capability, mathematical reasoning, and code generation benchmarks, where MONA achieves SOTA performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。