Muon优化器让模型更快实现从死记硬背到真正理解的转变。
Muon Optimizer Accelerates Grokking
- 用谱范数约束和二阶信息设计的Muon优化器，改进了学习路径。
- 在7个算术任务上，平均提前50.2个周期进入grokking阶段。
- 适合关注模型泛化能力提升与优化器设计的研究者。
本文研究不同优化器对模型延迟泛化（grokking）现象的影响。我们在7个数值任务（以模运算为主）上，使用现代Transformer架构进行实验，系统性地对比了优化器（Muon vs. AdamW）与softmax激活函数（标准softmax、stablemax、sparsemax）的组合效果。实证结果表明，采用谱范数约束和二阶信息的Muon优化器，显著加速了grokking的出现。相比广泛使用的AdamW，Muon将所有配置下的平均grokking周期从153.09降至102.89，差异具有统计显著性（t = 5.0175, p = 6.33e-08）。这说明优化器选择在推动模型从记忆走向泛化中起关键作用。
原文摘要 · Abstract (English)
This paper investigates the impact of different optimizers on the grokking phenomenon, where models exhibit delayed generalization. We conducted experiments across seven numerical tasks (primarily modular arithmetic) using a modern Transformer architecture. The experimental configuration systematically varied the optimizer (Muon vs. AdamW) and the softmax activation function (standard softmax, stablemax, and sparsemax) to assess their combined effect on learning dynamics. Our empirical evaluation reveals that the Muon optimizer, characterized by its use of spectral norm constraints and second-order information, significantly accelerates the onset of grokking compared to the widely used AdamW optimizer. Specifically, Muon reduced the mean grokking epoch from 153.09 to 102.89 across all configurations, a statistically significant difference (t = 5.0175, p = 6.33e-08). This suggests that the optimizer choice plays a crucial role in facilitating the transition from memorization to generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。