用动量方差减少改进深度学习优化器，收敛更快。
Muon is Provably Faster with Momentum Variance Reduction
- 将动量方差减少引入Gluon框架，统一优化多个非欧几里得方法。
- 非凸情况下收敛率从O(1/K^1/4)提升至O(1/K^1/3)。
- 适合大模型训练，尤其关注优化效率的研究者。
近期实证研究显示，基于线性最小化预言机（LMO）并选择特定非欧几里得范数球的优化器（如Muon和Scion），在大规模语言模型训练中优于Adam类方法。本文证明，通过用动量方差减少（MVR）替换其原始动量，可进一步提升这些优化器的性能。我们不分别设计Muon和Scion的MVR变体，而是将MVR融入最近提出的Gluon框架——该框架能涵盖Muon、Scion及其他基于非欧几里得LMO的方法，并支持更一般化的光滑性假设，更贴合神经网络的层结构。在非凸情形下，我们以三种方式引入MVR，均使收敛率从O(1/K^{1/4})提升至O(1/K^{1/3})。此外，在星凸情形下也获得了改进的收敛速率。最后，通过若干数值实验验证了所提算法在迭代复杂度上的优越性。
原文摘要 · Abstract (English)
Recent empirical research has demonstrated that deep learning optimizers based on the linear minimization oracle (LMO) over specifically chosen Non-Euclidean norm balls, such as Muon and Scion, outperform Adam-type methods in the training of large language models. In this work, we show that such optimizers can be provably improved by replacing their vanilla momentum by momentum variance reduction (MVR). Instead of proposing and analyzing MVR variants of Muon and Scion separately, we incorporate MVR into the recently proposed Gluon framework, which captures Muon, Scion and other specific Non-Euclidean LMO-based methods as special cases, and at the same time works with a more general smoothness assumption which better captures the layer-wise structure of neural networks. In the non-convex case, we incorporate MVR into Gluon in three different ways. All of them improve the convergence rate from ${\cal O} (\frac{1}{K^{1/4}})$ to ${\cal O} (\frac{1}{K^{1/3}})$. Additionally, we provide improved rates in the star-convex case. Finally, we conduct several numerical experiments that verify the superior performance of our proposed algorithms in terms of iteration complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。