提出低成本正交化方法,证明训练效果不依赖高精度正交分解。
How Much Orthogonalization Does Muon Need?

- 设计五步三次牛顿-舒尔迭代,仅需10次矩阵乘法替代原方法的15次。
- 在GPT-2 Small和百亿参数混合模型上,新方法损失与顶尖方案几乎无差别。
- 适合追求高效训练的工程师,尤其适用于资源受限的模型优化场景。
Muon优化器通过将病态动量更新替换为近似半正交更新来提升神经网络训练效率。本文探讨其正交化程度的实际需求:基于针对低精度奇异值带直接推导的松弛三次牛顿-舒尔调度,提出一种五步三次构造,仅需十次主导矩阵乘法,少于五次五次牛顿-舒尔迭代的十五次。该方法并非追求更高极分解精度,而是作为一种原理清晰、成本低廉的变体,用于探究极分解精度、谱形塑性与训练质量之间的关系。在合成诊断、NanoGPT消融实验以及混合MoE/Mamba模型(参数量10亿至40亿)的训练中发现,训练质量并非随极分解精度单调提升:截断极坐标表达式、Muon-Jordan、三次牛顿-舒尔及显式FP32 SVD极因子在GPT-2 Small上达到几乎无法区分的最终损失;三次5(cubic5)在混合模型上的验证损失与Muon-Jordan五次更新相差约10⁻³。这些结果支持cubic5作为实际可用的低成本Muon正交化方案,在测试设置下具有训练质量等效性。
原文摘要 · Abstract (English)
Muon optimizers improve neural-network training by replacing ill-conditioned momentum updates with approximately semi-orthogonal updates. This motivates a practical question: how much orthogonalization does Muon actually require? We study this question using a relaxed cubic Newton--Schulz schedule derived directly for Muon's low precision singular value band. The resulting five-step cubic construction uses ten dominant matrix multiplications, compared with fifteen for five quintic Newton--Schulz iterations. The cubic schedule is not intended as a more accurate polar solver; instead, it is a principled low-cost variant that lets us probe the relation between polar accuracy, spectral shaping, and training quality. Across synthetic diagnostics, NanoGPT ablations, and training experiments on hybrid MoE/Mamba models, we find that training quality is not governed monotonically by polar-decomposition accuracy: truncated Polar Express, Muon-Jordan, cubic Newton--Schulz, and an explicit FP32 SVD polar factor can reach nearly indistinguishable final loss on GPT-2 Small, and cubic5 matches the Muon-Jordan quintic update within about $10^{-3}$ validation loss on hybrid MoE/Mamba models with one billion to four billion parameters. These results support cubic5 as a practical low-cost Muon orthogonalization variant, with empirical evidence of training-quality parity in the settings tested.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。