改进大模型训练中的参数更新机制，提升性能并保持稳定性。
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
- 引入重尾谱修正机制，增强参数更新的多样性。
- 在LLaMA预训练中将困惑度降低0.98，优于现有基线。
- 可作为插件适配现有方法，适合大模型训练研究者使用。
Muon在大语言模型训练中展现出良好效果。本文指出，Muon的正交化更新规则抑制了重尾权重谱的出现，并过度强调噪声主导方向的训练。基于重尾自正则化理论，提出HTMuon，既保留参数相互依赖性建模能力，又生成更重尾的更新和权重谱。在大模型预训练与图像分类任务中，HTMuon持续优于先进基线，例如在C4数据集上对LLaMA预训练，困惑度降低高达0.98。理论分析表明，HTMuon对应于施瓦茨-$q$范数约束下的最速下降，并在光滑非凸设置下提供收敛性证明。代码已开源：https://github.com/TDCSZ327/HTmuon。
原文摘要 · Abstract (English)
Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and over-emphasizes the training along noise-dominated directions. Motivated by the Heavy-Tailed Self-Regularization (HT-SR) theory, we propose HTMuon. HTMuon preserves Muon's ability to capture parameter interdependencies while producing heavier-tailed updates and inducing heavier-tailed weight spectra. Experiments on LLM pretraining and image classification show that HTMuon consistently improves performance over state-of-the-art baselines and can also serve as a plug-in on top of existing Muon variants. For example, on LLaMA pretraining on the C4 dataset, HTMuon reduces perplexity by up to $0.98$ compared to Muon. We further theoretically show that HTMuon corresponds to steepest descent under the Schatten-$q$ norm constraint and provide convergence analysis in smooth non-convex settings. The implementation of HTMuon is available at https://github.com/TDCSZ327/HTmuon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。