通过谱正交化实现自适应预处理，让优化器在矩阵问题中更快收敛。
Preconditioning Benefits of Spectral Orthogonalization in Muon
- 用谱域正交化分解梯度，使优化过程变为独立标量序列迭代
- 在矩阵分解和线性Transformer上下文学习中，收敛速度与条件数无关
- 理论证明其优于梯度下降和Adam，适用于大模型预训练场景
Muon优化器是一种基于梯度谱正交化的矩阵结构算法，是大规模语言模型预训练的重要里程碑。然而，其内在机制——特别是梯度正交化的作用——仍不清晰，鲜有研究提供端到端的严谨分析来解释其在具体应用中的优势。本文通过两个案例研究：矩阵分解和线性Transformer的上下文学习，分析简化版Muon的有效性。我们证明，该简化版Muon在两种问题上均实现线性收敛，且迭代复杂度与相关条件数无关，显著优于梯度下降和Adam。理论分析揭示，Muon动态在谱域可解耦为一系列独立标量序列，具有相似收敛行为。该理论形式化了谱正交化带来的预处理效应，深入解释了Muon在这些矩阵优化问题中的有效性，并可能推广至更广泛场景。
原文摘要 · Abstract (English)
The Muon optimizer, a matrix-structured algorithm that leverages spectral orthogonalization of gradients, is a milestone in the pretraining of large language models. However, the underlying mechanisms of Muon -- particularly the role of gradient orthogonalization -- remain poorly understood, with very few works providing end-to-end analyses that rigorously explain its advantages in concrete applications. We take a step by studying the effectiveness of a simplified variant of Muon through two case studies: matrix factorization, and in-context learning of linear transformers. For both problems, we prove that simplified Muon converges linearly with iteration complexities independent of the relevant condition number, provably outperforming gradient descent and Adam. Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior. Our theory formalizes the preconditioning effect induced by spectral orthogonalization, offering insight into Muon's effectiveness in these matrix optimization problems and potentially beyond.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。