arXiv:2602.05725cs.LGmath.OC2026-02被引 7

Muon优化器加速记忆学习,突破传统梯度下降的频率瓶颈。

Muon in Associative Memory Learning: Training Dynamics and Scaling Laws

  • Muon通过梯度矩阵符号更新,实现频率成分的均衡学习
  • 无噪声时速度比梯度下降快指数级,有噪声时具更优缩放效率
  • 可解释为自适应任务对齐的隐式矩阵预处理,适合长尾任务

Muon通过梯度矩阵的符号更新参数,在实践中表现优异,但其理论动态与缩放规律尚不明确。本文在带softmax检索的线性关联记忆模型中研究了Muon,该模型具有查询-答案对的分层频率谱,并考虑了有无标签噪声的情况。结果表明,梯度下降(GD)以极不均衡的速度学习不同频率成分,导致收敛被低频成分拖慢;而Muon缓解了这一不平衡,实现更快且更均匀的学习进程。具体而言,在无噪声情况下,Muon相较GD实现指数级加速;在具有幂律频率谱的噪声场景中,我们推导出Muon的缩放律,并证明其优于GD。此外,我们发现Muon可视为由自适应任务对齐和块对称梯度结构产生的隐式矩阵预处理;而仅当能访问未知任务表示时,坐标系符号算子的预处理才可匹配Muon,这在实际SignGD中不可行。合成长尾分类与LLaMA风格预训练实验验证了理论结论。

原文摘要 · Abstract (English)

Muon updates matrix parameters via the matrix sign of the gradient and has shown strong empirical gains, yet its dynamics and scaling behavior remain unclear in theory. We study Muon in a linear associative memory model with softmax retrieval and a hierarchical frequency spectrum over query-answer pairs, with and without label noise. In this setting, we show that Gradient Descent (GD) learns frequency components at highly imbalanced rates, leading to slow convergence bottlenecked by low-frequency components. In contrast, the Muon optimizer mitigates this imbalance, leading to faster and more uniform progress. Specifically, in the noiseless case, Muon achieves an exponential speedup over GD; in the noisy case with a power-law frequency spectrum, we derive Muon's scaling law and demonstrate its superior scaling efficiency over GD. Furthermore, we show that Muon can be interpreted as an implicit matrix preconditioner arising from adaptive task alignment and block-symmetric gradient structure. In contrast, the preconditioner with coordinate-wise sign operator could match Muon under oracle access to unknown task representations, which is infeasible for SignGD in practice. Experiments on synthetic long-tail classification and LLaMA-style pre-training corroborate the theory.

优化算法关联记忆缩放定律长尾学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。