arXiv:2510.19933cs.LGmath.OC2025-10被引 26

分析了优化器Muon在实际应用中的近似更新问题,揭示精度与学习率的深层关联。

Beyond the Ideal: Analyzing the Inexact Muon Update

  • 引入加性误差模型,分析实际近似正交化带来的影响
  • 发现近似误差越大,需用更小学习率但更大动量
  • 实验验证精度变化会显著影响最优学习率选择

Muon优化器因其几何感知能力成为AdamW的有力替代,在大规模神经网络训练中表现优异。然而其效率依赖快速近似正交化,而现有理论分析均基于计算不可行的精确SVD更新。本文首次对核心的不精确正交化更新进行分析,基于线性最小化预言机(LMO)框架,构建真实可行的加性误差模型。分析给出了性能退化的显式边界,揭示了误差与最优步长、动量间的根本耦合关系:误差越大,需更小步长但更大动量。这表明近似过程(如牛顿-舒尔茨迭代次数)不再是实现细节,而必须与学习率调度协同调优。NanoGPT实验直接验证了该耦合关系,最优学习率随近似精度变化明显移动。

原文摘要 · Abstract (English)

The Muon optimizer has rapidly emerged as a powerful, geometry-aware alternative to AdamW, demonstrating strong performance in large-scale training of neural networks. However, a critical theory-practice disconnect exists: Muon's efficiency relies on fast, approximate orthogonalization, yet all prior theoretical work analyzes an idealized, computationally intractable version assuming exact SVD-based updates. This work moves beyond the ideal by providing the first analysis of the inexact orthogonalized update at Muon's core. We develop our analysis within the general framework of Linear Minimization Oracle (LMO)-based optimization, introducing a realistic additive error model to capture the inexactness of practical approximation schemes. Our analysis yields explicit bounds that quantify performance degradation as a function of the LMO inexactness/error. We reveal a fundamental coupling between this inexactness and the optimal step size and momentum: lower oracle precision requires a smaller step size but larger momentum parameter. These findings elevate the approximation procedure (e.g., the number of Newton-Schulz steps) from an implementation detail to a critical parameter that must be co-tuned with the learning schedule. NanoGPT experiments directly confirm the predicted coupling, with optimal learning rates clearly shifting as approximation precision changes.

优化器近似计算深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。