Muon通过谱平坦化实现稳定高速优化，突破传统学习率瓶颈。
Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
- 用牛顿-舒尔茨迭代对动量缓冲区正交化，使梯度谱更平坦
- 最大稳定步长与梯度平均奇异值相关，而非最大值，提升稳定性
- 适合需要大步长、快速收敛的深度学习训练场景
Muon 在每次更新前对动量缓冲区进行正交化，通过牛顿-舒尔茨迭代将其奇异值替换为1。这一简单改动使 Muon 能容忍远大于其他优化器的学习率并更快收敛，但其原理是什么？我们揭示其机制为谱平坦化，并提出两项结果：首先，证明了 Muon 的最大稳定步长与梯度的平均奇异值成正比，而非传统梯度下降所受最大奇异值的限制；其次，将 Muon 重述为预条件梯度方法，在克罗内克分解曲率模型下，其有效收敛因子得到改善，改进程度由梯度协方差谱控制。大量实验验证了这两项结论：在导致 SGD 在前几轮就发散的学习率下，Muon 仍保持稳定；即使在相同步长下，也提前数个训练周期达到精度里程碑。综上，我们的结果为 Muon 的经验成功提供了原理性、几何化的解释。
原文摘要 · Abstract (English)
Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。