Muon优化器微调时效果差，用LoRA可缓解性能下降
Can Muon Fine-tune Adam-Pretrained Models?
- 用LoRA限制更新幅度，缓解Adam与Muon的优化器不匹配
- 实验表明更新越强，性能下降越严重，与更新强度正相关
- 适合研究优化器兼容性或做模型微调的开发者参考
Muon作为预训练阶段的高效替代优化器，但在微调中应用较少。主要障碍在于多数开源模型使用Adam预训练，直接切换到Muon微调会导致性能下降，这是由于两者隐含偏差不同所致。通过控制实验发现，这种不匹配会破坏预训练知识，且破坏程度随更新强度增加而加剧。据此提出约束更新的策略，并通过LoRA验证：在语言和视觉任务中，LoRA显著缩小了全量微调下Adam与Muon的性能差距。对LoRA秩、灾难性遗忘及变体的研究进一步证实，不匹配严重性与更新强度密切相关。结果揭示了优化器不匹配的影响机制及其缓解方法。代码已公开于https://github.com/XingyuQu/muon-finetune。
原文摘要 · Abstract (English)
Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to an optimizer mismatch. We investigate this mismatch through controlled experiments and relate it to the distinct implicit biases of Adam and Muon. We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength. This leads us to hypothesize that constraining updates should mitigate the mismatch. We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning. Studies on LoRA rank, catastrophic forgetting, and LoRA variants further confirm that mismatch severity correlates with update strength. These results shed light on how optimizer mismatch affects fine-tuning and how it can be mitigated. Our code is available at https://github.com/XingyuQu/muon-finetune.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。