发现LoRA微调中奇异值均匀增长,揭示了新优化规律。
Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization
- 在LoRA结构下分析谱梯度流,发现奇异值等速增长。
- 小奇异值比大奇异值更早达到目标值,与传统学习相反。
- 理论证明在合理条件下全局收敛,适合理解大模型优化。
谱梯度下降(SpecGD)通过正交化矩阵参数更新,启发了如Muon等实用优化器,在大语言模型(LLM)训练中表现优异,但其动态机制仍不明确。在低秩适配(LoRA)设置中,权重更新被表示为两个低秩因子的乘积,我们发现:在使用Muon进行LLM微调时,尽管两个因子分别被正交化,其乘积的奇异值却呈现近似均匀增长。受此现象启发,我们在简化版的LoRA式矩阵分解设定中分析了谱梯度流(SpecGF)——SpecGD的连续时间版本,并证明了“等速”动力学:所有奇异值以几乎相等速率增长,存在微小偏差。因此,较小奇异值会比较大奇异值更早达到目标值,这与标准梯度流中‘最大优先’的分步学习形成鲜明对比。此外,我们证明在因子范数有界条件下,该设定下的SpecGF可从几乎所有初始点收敛至全局最小值;若加入ℓ₂正则化,则可获得全局收敛性。最后,我们在相同设定中通过实验验证了理论结果。
原文摘要 · Abstract (English)
Spectral gradient descent (SpecGD) orthogonalizes the matrix parameter updates and has inspired practical optimizers such as Muon. They often perform well in large language model (LLM) training, but their dynamics remain poorly understood. In the low-rank adaptation (LoRA) setting, where weight updates are parameterized as a product of two low-rank factors, we find a distinctive spectral phenomenon under Muon in LoRA fine-tuning of LLMs: singular values of the LoRA product show near-uniform growth across the spectrum, despite orthogonalization being performed on the two factors separately. Motivated by this observation, we analyze spectral gradient flow (SpecGF)-a continuous-time analogue of SpecGD-in a simplified LoRA-style matrix factorization setting and prove "equal-rate" dynamics: all singular values grow at equal rates up to small deviations. Consequently, smaller singular values attain their target values earlier than larger ones, sharply contrasting with the largest-first stepwise learning observed in standard gradient flow. Moreover, we prove that SpecGF in our setting converges to global minima from almost all initializations, provided the factor norms remain bounded; with $\ell_2$ regularization, we obtain global convergence. Lastly, we corroborate our theory with experiments in the same setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。