Muon优化器本质是训练中的隐式残差连接。
Muon as a Residual Connection
- 通过正交化更新，牺牲部分梯度精度以保留下游层可用的表示。
- 在线性优化中，Muon学习到的表征更难拟合局部目标但更利于下游使用。
- 为设计兼顾局部下降与下游可用性的优化器提供新思路。
Muon最近成为训练大型神经网络最有效的优化器之一，但其经验成功已被从多个不同角度解释。本文提出一种简单机制性解释：Muon在训练过程中可被理解为一种隐式残差连接。具体而言，对更新进行正交化会牺牲部分即时梯度保真度，但能改善下游层的表示保留。我们在受控的线性优化设置中研究这一权衡，发现Muon能够学习到更难拟合局部目标但更易被下游层利用的表征。结果表明，这为Muon提供了概念性解释，并为优化器设计提供了兼顾局部下降与下游可用性的新视角。
原文摘要 · Abstract (English)
Muon has recently emerged as one of the most effective optimizers for training large neural networks, yet its empirical success has been explained from several different perspectives. In this paper, we propose a simple mechanistic interpretation: Muon can be understood as an implicit residual connection during training. Specifically, orthogonalizing the update can sacrifice some immediate gradient fidelity while improving representation preservation for downstream layers. We study this trade-off in controlled linear optimization settings, where Muon can learn representations that are slower to fit a local target but easier for downstream layers to exploit. Our results suggest a conceptual explanation for Muon and a design perspective for optimizers that balance local descent with downstream usability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。