解析了新型优化器Muon的收敛性与最优批量大小,揭示其超参调控机制。
Convergence Bound and Critical Batch Size of Muon Optimizer
- 基于矩阵结构设计理论分析,覆盖带与不带动量/权重衰减场景
- 证明权重衰减可保证参数与梯度有界,无需梯度有界假设
- 推导出关键批量大小的下界,揭示动量与衰减系数的调控规律
Muon是一种新提出的优化器,利用神经网络参数的固有矩阵结构,在实践中表现出优异性能,显示其有望替代AdamW等标准优化器。本文提供理论分析以支持其实用成功:为四种实际场景下的Muon提供了收敛性证明,系统考察了是否引入Nesterov动量和权重衰减的影响。结果表明,权重衰减能确保参数和梯度范数几乎必然有界,且不依赖常见的梯度有界假设,并阐明了权重衰减系数与学习率之间的关系。最后,我们推导出使随机一阶预言机(SFO)复杂度最小化的临界批量大小的下界。由于公式涉及不可直接观测的问题相关量(梯度方差、目标精度、有效秩),无法给出绝对值预测,但揭示了超参数β(动量)和λ(权重衰减)对临界批量大小的定性调控作用。实验在图像分类与语言建模任务上验证了这些超参依赖的预测。
原文摘要 · Abstract (English)
Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW. This paper presents theoretical analysis to support its practical success. We provide convergence proofs for Muon across four practical settings, systematically examining its behavior with and without the inclusion of Nesterov momentum and weight decay. We then demonstrate that the addition of weight decay ensures almost-sure boundedness of the parameter and gradient norms -- without relying on the commonly imposed bounded-gradient assumption -- and clarify the interplay between the weight decay coefficient and the learning rate. Finally, we derive a lower bound on the critical batch size for Muon -- the batch size that minimizes the stochastic first-order oracle (SFO) complexity of training. Because the resulting formula involves problem-dependent quantities that are not directly observable (gradient variance, target precision, effective rank), it does not predict the critical batch size in absolute terms; rather, it reveals how the hyperparameters $β$ (momentum) and $λ$ (weight decay) govern the qualitative scaling of this value. Our experiments validate these hyperparameter-dependent predictions across workloads including image classification and language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。