arXiv:2606.21514cs.LG2026-06被引 2

Muon优化器在大模型训练中表现不稳定,研究揭示其优劣根源并建议分阶段使用。

Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective

论文配图:Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective
图 1 · 摘自论文原文
  • 从河流山谷视角分析优化路径,揭示Muon早期快但后期慢的机制
  • 在低谷区域因去相关更新导致震荡,收敛速度慢于梯度下降
  • 建议前期用Muon加速,后期切换为类似GD的优化器提升性能

近期,Muon作为Adam类优化器的替代方案受到广泛关注,其理论优势体现在谱归一化和条件改善上。然而,在大语言模型(LLM)训练中,Muon相对于Adam/AdamW的表现往往不一致、依赖学习率调度且非普遍更优。为弥合这一差距,我们提出轨迹级理论,刻画Muon的优势与局限。构建混合尖峰矩阵感知模型,其感知算子分解为信号、尖峰与主体成分,模拟了LLM训练中的各向异性结构与长尾信息特征。基于河流山谷视角,将优化景观视为流向目标解的河流方向与携带干扰信息的山丘方向的组合。在无动量情况下,我们发现Muon在优化早期沿信息主导的河流方向移动更快,但在接近河底时收敛远慢于梯度下降。进一步将该视角推广至带动量的非凸目标,研究谱河流上的点,结果显示:尽管早期收敛更快,但其正交化更新会丢失残余尺度信息,导致靠近目标时易发生超调与震荡。这些结果表明,我们的分析超越了尖峰矩阵感知模型,支持在训练后期切换至类似梯度下降的精修优化器,而非仅依赖固定学习率调度。我们在语言模型训练实验中提供了初步证据支持该两阶段策略。

原文摘要 · Abstract (English)

Recently, Muon has gained substantial attention as an appealing alternative to Adam-like optimizers, with many works highlighting its advantages through spectral normalization and improved conditioning. Yet this positive theoretical narrative contrasts with its empirical performance in large language model (LLM) training, where Muon's gains over Adam/AdamW are often mixed, schedule-sensitive, and not uniformly superior. To address this gap, we develop a trajectory-level theory characterizing both the strengths and limitations of Muon. We introduce a mixed-spiked matrix sensing model whose sensing operator decomposes into signal, spike, and bulk components, capturing a mixture of anisotropic structure and long-tail information reminiscent of LLM training. On top of it, we adopted a river-valley perspective in which we view the landscape as composed of a river direction flowing to the desired solution and hill directions encoding nuisance or task-irrelevant information. In the momentum-free setting, we show that Muon moves faster along the information-bearing river direction during early optimization, but can converge much more slowly near the river bottom than gradient descent. We then extend the river-valley perspective to general nonconvex objectives with momentum by studying points on the spectral river. There, while Muon converges faster early on, its orthogonalized update removes residual scale information, making it prone to overshooting and oscillation near the target solution. Together, these results suggest that our characterizations extend beyond spiked matrix sensing and motivate switching to GD-like refinement optimizers in the final phase, rather than relying only on a fixed learning-rate schedule for Muon. We also provide preliminary evidence supporting this two-stage approach in language model training experiments.

优化器大模型训练机器学习理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。