arXiv:2605.26459cs.LG2026-05

提出一种高效训练大模型的优化器剪裁方法,提升稳定性与计算效率。

MuCon: Clipped Muon Updates for LLM Training

  • 用奇异值剪裁替代传统极分解,简化更新方向计算。
  • 发现阈值附近奇异值导致数值不稳,影响优化精度。
  • 适合追求高效稳定训练的大模型研究者使用。

Muon类优化器采用矩阵动量或预条件更新 $B = U ext{diag}(σ_1, dots,σ_r) V^ op$,并替换为它的规范部分极因子 $ ext{Pol}(B) = U V^ op$,将所有非零奇异值映射为1。本文研究的MuCon是其剪裁变体:对同一矩阵应用奇异值剪裁,定义为 $D^{ ext{MuCon}}_τ(B) = ext{MClip}_τ(B) = U ext{diag}( ext{min}iglackslash{σ_i,τiglackslash}) V^ op$,其中 $τ>0$。$ ext{MClip}_τ$ 是数学上的剪裁算子,MuCon则是将此剪裁方向取代原极方向的优化器基元。本文采用的缩放参数化方式称为 $ ext{SpectralP}$,即在隐矩阵缩放下应用极方向或剪裁后的方向。$ ext{MClip}_τ$ 等价于Frobenius范数球 $\\{X : \|X\|_2 \le τ\\\\\\" 上的投影:保持不超过 $τ$ 的奇异值不变,仅修改超出的部分。本文探讨了能否避免完整稠密SVD来近似剪裁步骤。研究发现两个精确恒等式:极/绝对值公式与标量根公式,后者导出有理Newton滤波器用于剪裁后的半正定因子;但二者共同的数值障碍在于接近阈值的奇异值引发符号判断困难及有理求解病态。因此,矩阵函数方法仅在结合稳定极分解或平方根基元,或在剪裁边界附近引入显式正则化时才有效。

原文摘要 · Abstract (English)

Muon-style optimizers take a matrix-valued momentum or preconditioned update $B = U \operatorname{diag}(σ_1,\ldots,σ_r) V^\top$ and replace it with its canonical partial polar factor $\operatorname{Pol}(B) = U V^\top$. This maps every nonzero singular value to one. MuCon is the clipped-Muon variant studied here: it applies singular-value clipping to the same Muon matrix, $D^{\mathrm{MuCon}}\_τ(B) = \operatorname{MClip}\_τ(B) = U \operatorname{diag}\bigl(\min\{σ\_i,τ\}\bigr) V^\top, \qquad τ> 0$. Thus, $\operatorname{MClip}\_τ$ denotes the mathematical clipping operator, while MuCon denotes the optimizer primitive that substitutes this clipped direction for Muon's polar direction. The Muon/MuCon scaling parameterization used in this work is called $\text{SpectralP}$: it is the hidden-matrix scaling recipe under which polar Muon or clipped MuCon directions are applied. The map $\operatorname{MClip}\_τ$ is the Frobenius projection onto the spectral-norm ball $\{X : \|X\|_2 \le τ\}$: it leaves singular values at or below $τ$ unchanged and modifies only the violating singular directions. This paper asks when the MuCon clipping step can be approximated without a full dense SVD. We record two exact identities, a polar/absolute-value formula and a scalar-root formulation leading to a rational Newton filter for the clipped positive-semidefinite factor, and identify the numerical obstruction common to both: singular values near the threshold make sign decisions and rational solves ill-conditioned. Matrix-function methods are therefore useful only when paired with stable polar/square-root primitives or explicit regularization near the clipping boundary.

优化器大模型剪裁数值稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。