MuonAll让大模型微调更高效，性能媲美AdamW
MuonAll: Muon Variant for Efficient Finetuning of Large Language Models
- 将优化器参数全转为二维矩阵，实现全局优化
- 在百亿参数以下模型上表现与AdamW相当
- 适合需要高效微调的NLP研究者和工程师
Muon优化器在语言模型预训练中表现稳健，但其在现有公开预训练模型微调中的效果尚未探索。当前使用时通常搭配AdamW，存在改进空间。本文提出MuonAll，通过将所有参数转换为二维矩阵，实现完全集成于Muon的优化策略。我们在多个公开语言模型上进行广泛微调实验，模型规模达5亿参数以内。结果表明，Muon与MuonAll在主流基准测试中表现与AdamW相当，证明其作为替代优化器的有效性。我们已开源分布式实现，项目地址：https://github.com/Saurabh750/optimizer。
原文摘要 · Abstract (English)
Muon optimizer has demonstrated robust results in pretraining of language models but its performance in finetuning of existing public pretrained models is not yet explored. Currently, Muon is used along with AdamW introducing a scope of improvement for adopting all parameters inside Muon. We introduce MuonAll, which incorporates all the parameters inside Muon by transforming into 2D matrices. We conduct extensive finetuning experiments across publicly available language models with model sizes upto half billion parameters. Muon and MuonAll perform at par with AdamW across major benchmarks, highlighting their effectiveness as alternative optimizers. We open-source the distributed implementations of Muon and MuonAll, available at https://github.com/Saurabh750/optimizer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。