Muon优化器经改进后可高效训练大模型，显著提升计算效率。
Muon is Scalable for LLM Training
- 通过加入权重衰减和调整参数更新尺度，实现大模型零调参训练。
- 相比AdamW，Muon在最优训练下提升约2倍计算效率，节省大量算力。
- 开源了高效分布式实现及多个版本模型，支持后续研究与应用。
基于矩阵正交化的Muon优化器在小规模语言模型训练中表现优异，但其在大规模模型上的可扩展性尚未验证。本文识别出两项关键改进：（1）引入权重衰减，（2）精细调节每参数更新尺度。这两项技术使Muon可在无需超参数调优的情况下直接应用于大规模训练。基于缩放定律实验，Muon在计算最优训练下相较AdamW实现约2倍的计算效率提升。在此基础上，我们推出了Moonlight——一个使用Muon训练的30亿和160亿参数的专家混合（MoE）模型，训练数据达5.7万亿词元。该模型突破现有性能-算力权衡边界，在更少训练浮点运算量下达到更高性能。我们开源了内存最优、通信高效的分布式Muon实现，并发布预训练、指令微调及中间检查点，以支持未来研究。
原文摘要 · Abstract (English)
Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves $\sim\!2\times$ computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。