提出sMuon方法，让低秩微调适配更高效的Muon优化器。
Approximate Muon with low-rank adapters
- 通过线性化与最小二乘法近似求解低秩下的Muon目标函数。
- 在SFT和ReLoRA预训练中表现优于基线，带来适度性能提升。
- 仅用矩阵乘法实现，计算高效，适合资源受限场景。
Muon优化器在神经网络预训练中表现优异，但在参数高效微调（PEFT）中应用较少。主要原因在于主流方法LoRA无法与Muon自然结合，因低秩参数化下无法实现权重更新的正交化。本文提出sMuon（小Muon），通过线性化和最小二乘法近似求解松弛后的Muon目标，在低秩设置下实现有效近似。该方法仅依赖矩阵乘法，无需复杂线性代数分解，计算效率高。在监督微调（SFT）和ReLoRA预训练实验中，sMuon表现良好，整体带来适度性能提升，证明将Muon用于低秩微调具有实际价值。
原文摘要 · Abstract (English)
The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses matmul operations only, as opposed to more complex linear algebra decomposition routines. Our method, sMuon (small Muon), performs favourably across SFT and a ReLoRA pretraining experiment. While results are model- and eval-dependent, we find overall that using Muon for low-rank fine-tuning provides moderate performance improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。