Muon通过平滑更新谱来减少优化偏差,提升多方向活跃时的训练稳定性。
The Spectral Dynamics and Noise Geometry of Muon

- 用极分解替代梯度矩阵,使更新谱趋于平坦。
- 在回归模型中证明谱趋向均匀分布,且与核范数最小化相反。
- 适合需多方向同时活跃的场景,如小规模GPT训练。
Muon将梯度矩阵 $G=UΣV^ op$ 替换为其极因子 $UV^ op$,保留奇异方向但使更新谱平坦。在显式对齐假设下,我们证明极更新是在使用梯度奇异方向、不自适应当前权重谱且受范数约束的更新中,一步熵最大化的选择。在欠定回归模型中,我们推导出连续时间下Muon的精确奇异值动态,并识别出一个依赖测量条件,在该条件下归一化谱趋向相等的非零奇异值。这一几何结构排除了常见的低秩解释:在固定Frobenius范数下,Muon的最优状态具有平坦谱,而核范数最小化则偏好谱集中。控制矩阵感知实验分离了该效应与简单梯度缩放的影响,显示范数匹配的梯度下降无法复现Muon行为,并在广泛消融实验中恢复了预测的谱平坦趋势。在小型NanoGPT预训练中,Muon保持稳定秩,拥有宽学习率平台期,并相比AdamW提升验证损失;而在匹配的小型ViT控制实验中,排名反转。结果表明其效果具有情境依赖性:并非普遍优越,但在多个谱方向需保持活跃时,其平坦谱偏差可提供帮助。
原文摘要 · Abstract (English)
Muon replaces a matrix gradient $G=UΣV^\top$ by its polar factor $UV^\top$. This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We study the optimization bias created by this operation. Under explicit alignment assumptions, we prove that the polar update is the one-step entropy-maximizing choice among bounded updates that use the gradient singular directions and do not adapt to the current weight spectrum. In an underdetermined regression model, we derive exact singular-value dynamics for continuous-time Muon and identify a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. This geometry also rules out a common low-rank interpretation: at fixed Frobenius norm, Muon's distinguished state has a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments separate the effect from simple gradient rescaling, show that norm-matched gradient descent does not reproduce Muon, and recover the predicted flattening trend across broad ablations. In small NanoGPT pretraining, Muon preserves stable rank, has a broad learning-rate plateau, and improves validation loss relative to AdamW; in a matched small-ViT control, the ranking reverses. The resulting picture is regime-dependent: Muon is not universally superior, but its flat-spectrum bias can help when many spectral directions need to remain active.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。