通过约束更新方向的核范数,提升大模型权重压缩性。
NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
- 在Muon优化器基础上加入核范数约束,引导权重向低秩结构演化。
- 在十亿参数级别模型上,压缩后模型性能显著优于基线方法。
- 适合需要高效部署的大模型应用,兼顾训练速度与压缩效果。
大语言模型(LLM)的快速发展受到内存和部署成本的制约,推动了压缩方法的应用。当前主流压缩方案利用训练后权重矩阵的低秩特性,该特性常与Adam等优化器的性质相关。最近提出的Muon优化器通过全秩更新步骤提升预训练效率,但其生成的权重空间结构尚未被充分研究。本文发现一个意外现象:尽管采用全秩更新,Muon训练出的模型仍呈现显著的低秩结构,可被标准压缩流程有效压缩。基于此,我们提出NuMuon,通过在更新方向上施加核范数约束,进一步强化权重的低秩性。在十亿参数级模型上,NuMuon在保持良好收敛性的前提下,提升了权重压缩率,并改善了压缩后的模型质量,适用于当前最先进的大模型压缩流水线。
原文摘要 · Abstract (English)
The rapid progress of large language models (LLMs) is increasingly constrained by memory and deployment costs, motivating compression methods for practical deployment. Many state-of-the-art compression pipelines leverage the low-rank structure of trained weight matrices, a phenomenon often associated with the properties of popular optimizers such as Adam. In this context, Muon is a recently proposed optimizer that improves LLM pretraining via full-rank update steps, but its induced weight-space structure has not been characterized yet. In this work, we report a surprising empirical finding: despite imposing full-rank updates, Muon-trained models exhibit pronounced low-rank structure in their weight matrices and are readily compressible under standard pipelines. Motivated by this insight, we propose NuMuon, which augments Muon with a nuclear-norm constraint on the update direction, further constraining the learned weights toward low-rank structure. Across billion-parameter-scale models, we show that NuMuon increases weight compressibility and improves post-compression model quality under state-of-the-art LLM compression pipelines while retaining Muon's favorable convergence behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。