Muon优化器用牛顿-舒尔法近似正交化，理论证明其收敛速度接近精确解。
Convergence of Muon with Newton-Schulz
- 用牛顿-舒尔迭代近似动量矩阵正交化，替代传统SVD方法
- 收敛速度与精确极分解相当，误差因子随迭代步数双指数下降
- 相比SGD动量，避免了秩的平方根损失，适合大规模优化
我们分析了原始且实际使用的Muon优化器：通过少量牛顿-舒尔步骤实现动量正交化。此前理论工作以基于SVD的极分解取代此关键步骤。本文证明，使用牛顿-舒尔的Muon在给定q步下，收敛至驻点的速度与基于SVD的极分解理想化版本一致，仅相差一个常数因子。我们进一步分析该因子，证明其随q呈双指数趋近于1，并随牛顿-舒尔中多项式阶数提升而改善。此外，我们证明Muon可消除相比向量级优化器（如SGD with momentum）典型的平方根秩损失。结果解释了为何少数低阶牛顿-舒尔步骤即可达到精确极分解表现，且耗时更少，阐明了牛顿-舒尔正交化对向量级优化器的优势。整体上，本理论验证了实践中的牛顿-舒尔设计，缩小了理论与实践的差距。
原文摘要 · Abstract (English)
We analyze Muon as originally proposed and used in practice -- using the momentum orthogonalization with a few Newton-Schulz steps. The prior theoretical results replace this key step in Muon with an exact SVD-based polar factor. We prove that Muon with Newton-Schulz converges to a stationary point at the same rate as the SVD-polar idealization, up to a constant factor for a given number $q$ of Newton-Schulz steps. We further analyze this constant factor and prove that it converges to 1 doubly exponentially in $q$ and improves with the degree of the polynomial used in Newton-Schulz for approximating the orthogonalization direction. We also prove that Muon removes the typical square-root-of-rank loss compared to its vector-based counterpart, SGD with momentum. Our results explain why Muon with a few low-degree Newton-Schulz steps matches exact-polar (SVD) behavior at a much faster wall-clock time and explain how much momentum matrix orthogonalization via Newton-Schulz benefits over the vector-based optimizer. Overall, our theory justifies the practical Newton-Schulz design of Muon, narrowing its practice-theory gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。