提出改进的动量优化器，理论证明其收敛速度达到最优。
On the Convergence of Muon and Beyond
- 设计双批次方差缩减变体，提升收敛性。
- 在非凸随机设置下实现 $\widetilde{\mathcal{O}}(T^{-1/3})$ 最优收敛率。
- 实验验证在CIFAR-10和C4上有效，适合追求理论性能的开发者。
Muon优化器在处理神经网络中的矩阵结构参数时表现出显著的实证效果，但其理论理解仍不充分。现有分析表明，Muon变体在随机非凸环境下仅能达到次优的遍历收敛率 $\mathcal{O}(T^{-1/4})$，其中 $T$ 为迭代次数。本文研究了两种基于动量的方差缩减变体：单批次 Muon-MVR1 和双批次 Muon-MVR2。在无时间窗学习率调度下，首次严格证明，方差缩减使 Muon-MVR2 达到最优的任意时刻收敛率 $\widetilde{\mathcal{O}}(T^{-1/3})$，与该问题类的下界一致。在 Polyak--Łojasiewicz (PL) 条件下，我们建立了任意时刻保证：Muon-MVR1 和 Muon-MVR2 分别实现期望平方根偏差的 $\widetilde{\mathcal{O}}(T^{-1/4})$ 与 $\widetilde{\mathcal{O}}(T^{-1/3})$ 最优率；在额外均匀梯度有界条件下，其最终迭代点分别达到 $\mathcal{O}(T^{-1/4})$ 与 $\mathcal{O}(T^{-1/3})$ 的目标间隙率。在 CIFAR-10 与 C4 上的实验支持所提方法的有效性。代码已开源于 Muon-MVR Codebase。
原文摘要 · Abstract (English)
The Muon optimizer has demonstrated remarkable empirical success in handling matrix-structured parameters for training neural networks. However, a significant gap remains between its practical performance and theoretical understanding. Existing analyses show that the Muon variants achieve only a suboptimal ergodic convergence rate of $\mathcal{O}(T^{-1/4})$ in stochastic non-convex settings, where $T$ denotes the number of iterations. To study the theoretical limits of Muon, we analyze two momentum-based variance-reduced variants: the one-batch Muon-MVR1 and the two-batch Muon-MVR2. We provide the first rigorous proof that, under \textbf{horizon-free} learning-rate schedules, variance reduction enables Muon-MVR2 to attain the optimal anytime convergence rate $\widetilde{\mathcal{O}}(T^{-1/3})$, matching the lower bound for this problem class. Under the Polyak--Łojasiewicz (PL) condition, we establish anytime guarantees for Muon-MVR1 and Muon-MVR2: they attain best-iterate rates of $\widetilde{\mathcal{O}}(T^{-1/4})$ and $\widetilde{\mathcal{O}}(T^{-1/3})$ for the expected square-root suboptimality, and, given an additional uniform gradient bound along the iterates, achieve last-iterate rates of $\mathcal{O}(T^{-1/4})$ and $\mathcal{O}(T^{-1/3})$ for the objective gap, respectively. Experiments on CIFAR-10 and C4 support the practical effectiveness of the proposed variance-reduced Muon variants. Code is available at \href{https://github.com/MaeChd/MUON-MVR}{Muon-MVR} Codebase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。