将方差缩减与矩阵预处理结合，提升大模型训练效率
MARS-M: When Variance Reduction Meets Matrices
- 融合MARS方差缩减与Muon矩阵预处理，优化训练过程
- 理论收敛速度达T^{-1/3}，优于Muon的T^{-1/4}
- 在语言与视觉任务中均实现更低损失和更好性能
基于矩阵的预处理优化器（如Muon）已被证明在训练大规模神经网络（包括大语言模型）时比标量优化器更高效。近期对大语言模型预训练优化器的基准研究表明，方差缩减技术（如MARS）相比未采用该技术的标准优化器可显著加速训练。本文提出MARS-M，将MARS风格的方差缩减与Muon相结合。在标准正则条件下，我们证明MARS-M以 ilde{ ext{O}}(T^{-1/3})的速率收敛至一阶驻点，优于Muon的 ilde{ ext{O}}(T^{-1/4})。在语言建模与计算机视觉任务上的实验表明，MARS-M在各类下游基准上持续取得更低损失与更优表现。MARS-M的实现代码已公开于https://github.com/AGI-Arena/MARS/tree/main/MARS_M。
原文摘要 · Abstract (English)
Matrix-based preconditioned optimizers, such as Muon, have recently been shown to be more efficient than scalar-based optimizers for training large-scale neural networks, including large language models (LLMs). Recent benchmark studies of LLM pretraining optimizers have demonstrated that variance-reduction techniques such as MARS can substantially speed up training compared with standard optimizers that do not employ variance reduction. In this paper, we introduce MARS-M, a new optimizer that integrates MARS-style variance reduction with Muon. Under standard regularity conditions, we prove that MARS-M converges to a first-order stationary point at a rate of $\tilde{\mathcal{O}}(T^{-1/3})$, improving upon the $\tilde{\mathcal{O}}(T^{-1/4})$ rate attained by Muon. Empirical results on language modeling and computer vision tasks demonstrate that MARS-M consistently yields lower losses and improved performance across various downstream benchmarks. The implementation of MARS-M is available at https://github.com/AGI-Arena/MARS/tree/main/MARS_M.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。