Muon优化器隐含约束权重矩阵谱范数，理论揭示其正则化机制。
Muon Optimizes Under Spectral Norm Constraints
- 将Muon归入Lion-K家族，用核范数解释其优化机制。
- 证明Muon在解耦权重衰减下隐含约束权重谱范数。
- 为设计新优化算法提供理论框架，适合研究优化器的学者。
加速优化算法的研究仍是深度学习的重要方向。近期提出的Muon优化器[JJB+24]展现出良好的实验性能，但其理论基础尚不清晰。本文将其置于Lion-\mathcal{K}优化器族[CLLL24]的框架下进行分析，表明当使用核范数时，Muon即对应于Lion-\mathcal{K}。基于Lion-\mathcal{K}的理论结果，我们证明了（采用解耦权重衰减的）Muon隐式求解一个对权重矩阵谱范数施加约束的优化问题。这一视角不仅揭示了Muon的隐式正则化作用，还通过改变凸映射\mathcal{K}的选取，自然推广出一类更广泛的隐式正则化与约束优化算法。
原文摘要 · Abstract (English)
The pursuit of faster optimization algorithms remains an active and important research direction in deep learning. Recently, the Muon optimizer [JJB+24] has demonstrated promising empirical performance, but its theoretical foundation remains less understood. In this paper, we bridge this gap and provide a theoretical analysis of Muon by placing it within the Lion-$\mathcal{K}$ family of optimizers [CLLL24]. Specifically, we show that Muon corresponds to Lion-$\mathcal{K}$ when equipped with the nuclear norm, and we leverage the theoretical results of Lion-$\mathcal{K}$ to establish that Muon (with decoupled weight decay) implicitly solves an optimization problem that enforces a constraint on the spectral norm of weight matrices. This perspective not only demystifies the implicit regularization effects of Muon but also leads to natural generalizations through varying the choice of convex map $\mathcal{K}$, allowing for the exploration of a broader class of implicitly regularized and constrained optimization algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。