arXiv:2608.26288cs.LGmath.OC2026-08

有限牛顿-舒尔茨迭代让优化器在非光滑非凸问题中更稳定收敛。

Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization

  • 用在线学习转换法分析优化器,将动量更新视为在线学习过程。
  • 有限牛顿-舒尔茨迭代可使奇异值映射平滑,实现对驻点的收敛。
  • 适用于大模型训练中的矩阵参数优化,尤其适合非光滑非凸场景。

Muon 已成为大型语言模型预训练中矩阵参数优化的有力工具,通过少量牛顿-舒尔茨迭代近似正交化其动量。现有理论要么用精确极分解替代该迭代,要么将有限深度视为近似误差,因而实际运行的有限迭代可能损害理论保证。本文证明,有限牛顿-舒尔茨迭代反而有助于非光滑非凸优化。我们通过在线学习到非凸优化的转换方法,将更新规则视为在线学习者,并将其后悔界转化为驻点保证。有限牛顿-舒尔茨将不连续的极映射平滑为奇异值的 Lipschitz 映射,使 Muon 可视为具有平滑谱势的在线学习者。这种平滑正是转换所需:我们证明,牛顿-舒尔茨深度仅需随目标精度对数增长,即可在非光滑非凸优化中收敛至驻点;而使用精确极分解的 Muon 可能无法收敛。所得样本复杂度界达到非光滑非凸优化最优水平,且对光滑非凸优化也近乎最优。该分析可推广至具相同平滑特性的通用谱映射。

原文摘要 · Abstract (English)

Muon has emerged as a strong optimizer for the matrix-valued parameters in large language model pretraining, approximately orthogonalizing its momentum with a few Newton-Schulz iterations. Existing theory either replaces this iteration with the exact polar factor it approximates, or treats its finite depth as an approximation error, and thus the iteration Muon actually runs can only hurt the guarantees. We show that finite Newton-Schulz can instead be beneficial for nonsmooth nonconvex optimization. To this end, we analyze Muon through the online-to-nonconvex conversion, which views the update rule as an online learner and converts its regret bound into a stationarity guarantee. The finite Newton-Schulz iteration smooths the discontinuous polar map into a Lipschitz map of the singular values, and Muon with finite Newton-Schulz can be regarded as an online learner with a smoothed spectral potential. This smoothing is exactly what the conversion needs: we prove that a Newton-Schulz depth growing only logarithmically in the target accuracy suffices for convergence to stationary points in nonsmooth nonconvex optimization, whereas Muon with the exact-polar update may fail to converge. The resulting sample complexity bounds match the best-known guarantees for nonsmooth nonconvex optimization and are optimal for smooth nonconvex optimization up to problem-dependent factors. The argument extends beyond Newton-Schulz to general spectral maps with the same smoothing property.

优化算法非凸优化矩阵优化牛顿-舒尔茨

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。