arXiv:2607.20512cs.LGcs.AI2026-07被引 1

Muon优化器加速模运算泛化,关键在正交化而非谱约束

The Active Ingredient in Muon's Grokking

  • 通过正交化迭代实现更快泛化,仅保留正交化即可复现加速效果
  • 正交化使模型在3倍更低的谱范数下达到泛化,收敛于低范数解
  • 单次正交化虽快但不稳定,五次迭代在不同学习率下最鲁棒

Muon优化器在模运算任务上比AdamW更快达到grokking阈值。以往研究归因于‘谱范数约束+正交化动量’,但未分离具体机制。我们通过多种子与多学习率实验分解并压力测试该效应:首先消融实验显示,加速来自正交化(Newton-Schulz迭代)——仅正交化即匹配完整Muon性能,而仅谱约束则不快于AdamW且不可靠,且结果在不同学习率下一致;其次机制分析发现,正交化优化器在约3倍更低的谱范数下实现泛化,且在控制嵌入实际移动量后,收敛于更低范数解而非仅减少扰动;第三,将Newton-Schulz迭代次数从5次减至1次虽加速达阈值,但导致解脆弱、易瞬时崩溃,脆弱性随学习率上升;单次迭代仅在小学习率下稳定,而标准5次迭代为学习率鲁棒的最优选择。此外,我们还表明谱缩放可完全移除而不影响性能。方法论主线贯穿始终:基于稳定性度量,'更快'的结论可能反转,因此同时报告首次跨越与持续泛化时间。为支持可复现性,我们已公开全部训练与分析代码(https://github.com/louiswang524/muon-grokking-frontier)。

原文摘要 · Abstract (English)

The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW. Prior work attributes this to "spectral-norm constraints plus orthogonalized momentum" but does not isolate which mechanism matters. To better understand Moun's behavior, we run multi-seed and multi-learning-rate sweeps to decompose and stress-test the effect. First, an ablation shows the speedup comes from orthogonalization (the Newton-Schulz iteration): orthogonalize-only matches full Muon, whereas spectral-only is no faster than AdamW and is unreliable, and this verdict holds across learning rates. Second, a mechanistic analysis finds that orthogonalizing optimizers reach generalization at roughly 3x lower spectral norm and, controlling for how much the embedding actually moves, settle into a lower-norm solution rather than simply perturbing the embedding less. Third, reducing the Newton-Schulz iteration count from five to one accelerates reaching the threshold but makes the grokked solution fragile, prone to transient collapse, with fragility that grows with learning rate; a single iteration is fast and stable only at small learning rate, while the canonical five iterations are the learning-rate-robust choice. We also show spectral scaling can be dropped at no measured cost. A methodological thread runs throughout: under a stability-aware metric, "faster" claims about grokking optimizers can invert, so we report both first-crossing and sustained-grok times. To support reproducibility, we release our full training and analysis code at https://github.com/louiswang524/muon-grokking-frontier

优化器泛化正交化Grokking

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。