arXiv:2605.17109cs.LGcs.AI2026-05被引 1

动态调整更新方向,让大模型训练更快更稳。

DynMuon: A Dynamic Spectral Shaping View of Muon

  • 用可变参数动态调整梯度更新方向,适应不同训练阶段。
  • 比原版Muon少10.6%-26.5%训练步数,达到同样验证损失。
  • 适合追求高效训练的大模型研究者与工程师。

近年来,Muon已成为训练大语言模型及更广泛Transformer架构的主流方法。其核心在于将标准梯度下降中的更新矩阵 $M=UΣV^ op$ 替换为极分解因子 $UV^ op$。本文提出一类类Muon更新:将 $M$ 替换为 $UΣ^p V^ op$,称为「谱形塑」操作,并建立理论指导如何根据(a)损失函数局部曲率、(b)随机梯度与标签噪声、(c)训练阶段选择参数 $p$。实验揭示新现象:正 $p$ 在早期强化高曲率方向,加速信号收缩;轻微负 $p$ 在后期将更新力重新分配至仍含有效信号的低曲率方向。基于此,我们提出DynMuon——一种在训练过程中从正到轻微负动态调度 $p$ 的高效方法。跨模型规模、架构与训练设置的大量实验表明,DynMuon持续优于Muon,在相同目标损失下减少10.6%-26.5%训练步数。代码已开源:https://github.com/fzwark/DynMuon。

原文摘要 · Abstract (English)

In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly. The essential difference, when compared to standard gradient descent methods, is to replace the usual update matrix $M=UΣV^\top$ with its polar factor $UV^\top$. In this work, we consider a class of Muon-like updates, where we replace the update $M$ with $UΣ^p V^\top$ for some parameter $p$. We call this a "spectral-shaping" operation, and develop a theory of how to pick $p$ which depends on (a) local curvature of the loss function, (b) noise stemming from stochastic gradients and label noise, and (c) training stage. Our theory and experimentation reveal a previously overlooked behavior: positive $p$ helps early by emphasizing high-curvature directions and accelerating signal contraction, while mildly negative $p$ helps later by reallocating update strength toward low-curvature directions that still contain useful training signals. Building on the insight, we propose DynMuon, an efficient dynamic spectral shaping method that schedules $p$ from positive to mildly negative over training. Extensive experiments across model sizes, architectures, and training settings show that DynMuon consistently achieves lower validation loss than Muon, while requiring 10.6-26.5% fewer steps to reach the same target loss. Our code is available at https://github.com/fzwark/DynMuon.

优化器大模型训练动态调整

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。