从谱视角解析Muon优化器,发现其本质是谱归一化而非万能解。
Delving into Muon and Beyond: Deep Analysis and Extensions
- 将Muon视为谱变换的p=0特例,拓展到p=1/4、1/2、1等变体
- 实验表明RMS归一化比一阶动量更稳定,Muon不总优于Adam
- 提出高效耦合牛顿迭代避免奇异值分解,适合深度学习优化研究者
Muon优化器因其出色的实证性能和对矩阵参数使用正交化更新而受到广泛关注,但其内在机制及与Adam等自适应优化器的关系仍不清晰。本文通过统一的谱视角进行研究,将Muon视为形式为UΣ^p V' 的谱变换族在p=0时的极限,并考察p=1/4、1/2、1等变体。该变换应用于一阶动量更新(如动量SGD)和均方根(RMS)归一化梯度更新(如Adam)。为实现高效计算,我们设计了一种避免显式奇异值分解的耦合牛顿迭代方法。在受控实验中,发现RMS归一化更新比一阶动量更新更具稳定性;尽管谱压缩在前者中带来显著稳定优势,但Muon(p=0)并未持续优于Adam。结果表明,Muon应被理解为一种有效的谱归一化形式,而非普遍更优的优化方法。代码将于https://github.com/Ocram7/BeyondMuon发布。
原文摘要 · Abstract (English)
The Muon optimizer has recently attracted considerable attention for its strong empirical performance and use of orthogonalized updates on matrix-shaped parameters, yet its underlying mechanisms and relationship to adaptive optimizers such as Adam remain insufficiently understood. In this work, we aim to address these questions through a unified spectral perspective. Specifically, we view Muon as the p = 0 endpoint of a family of spectral transformations of the form U \boldsymbolΣ^{p} V' , and consider additional variants with p = 1/2 , p = 1/4 , and p = 1 . These transformations are applied to both first-moment updates, as in momentum SGD, and to root-mean-square (RMS) normalized gradient updates as in Adam. To enable efficient computation, we develop a coupled Newton iteration that avoids explicit singular value decomposition. Across controlled experiments, we find that RMS-normalized updates yield more stable optimization than first-moment updates. Moreover, while spectral compression provides strong stabilization benefits under first-moment updates, the Muon update (p = 0) does not consistently outperform Adam. These results suggest that Muon is best understood as an effective form of spectral normalization, but not a universally superior optimization method. Our source code will be released at https://github.com/Ocram7/BeyondMuon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。