揭示动量在Muon中通过谱滤波提升训练稳定性
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
- 动量作为谱滤波器，抑制梯度噪声并保留主要信号
- 使正交化前的矩阵谱距扩大，稳定奇异子空间
- 先动量后正交优于反序或无动量，适合大规模语言模型训练
Muon 在大语言模型训练中表现优异，但其动量的理论作用仍不明确。现有分析要么移除动量以孤立研究谱更新，要么保留动量却无法解释为何提升性能。本文揭示：在结构化信号加扰动的梯度模型下，动量起到谱滤波作用，抑制扰动、保留主导信号，从而扩大两者间的谱距。该谱距扩大稳定了传入正交化步骤的矩阵的奇异子空间，使更新更可靠。进一步证明，先应用动量再正交化，比逆序或直接去除动量能更优对齐梯度信号成分。多项实验（包括大语言模型预训练）验证了理论。本理论为理解其他基于矩阵的优化器中动量的优势提供了起点。
原文摘要 · Abstract (English)
Muon has recently demonstrated strong empirical performance in large language model training, but the theoretical role of momentum in Muon remains unclear. Existing analyses of Muon either remove momentum to study spectral updates in isolation, or retain momentum without explaining why it improves empirical performance. Our work bridges this gap by showing momentum in Muon acts as a spectral filter. Under a structured signal-plus-perturbation gradient model, we prove that momentum suppresses perturbations while preserving the dominant signal, thereby enlarging the spectral gap between them. This enlarged gap stabilizes the singular subspaces of the matrix passed to Muon's orthogonalization step, making the resulting update more reliable. We further show that applying momentum before orthogonalization achieves provably stronger alignment with the signal component of the gradient than either reversing this order or simply removing momentum. Experiments across diverse tasks, including LLM pretraining, support our theoretical analysis. More broadly, our theory offers a starting point for understanding the benefits of momentum in other matrix-based optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。