揭示大模型优化中动量矩阵奇异值的缩放规律，指导高效计算。
Spectral Scaling Laws of Muon
- 通过追踪不同规模模型各层动量矩阵的奇异值分布，发现其在训练后趋于稳定。
- 奇异值稳定值随模型大小呈幂律变化，部分深层层缩放剧烈（可达 $M^{-0.96}$）。
- 为不同层提供可计算的NS迭代次数建议，避免冗余开销，适合大规模训练实践者。
正交化更新规则已成为训练大语言模型的主流选择，近期开源的顶尖模型已采用 Muon。为保持更新可计算性，Muon 使用牛顿-舒尔茨（NS）迭代进行正交化，但该方法仅为近似，小奇异值方向难以正交化。当前 Muon 在每一步对动量矩阵应用 NS 迭代，然而关于这些动量矩阵奇异值谱在训练过程中的演化，以及随模型规模的变化仍不明确。本文首次系统研究此问题，跟踪从 77M 到 2.8B 参数模型中各层动量缓冲区的奇异值分位数，发现经过短暂预热期后，分位数趋于稳定，且稳定值由层类型与模型规模决定。这些稳定值在模型规模上遵循清晰的幂律关系，层间指数不同。浅至中深层约以 $M^{-0.25}$ 缓慢变化，因此学术级常用的 5 步 NS 配置在更大规模下仍适用；而部分深层则更剧烈地缩放（最高达 $M^{-0.96}$），在前沿规模下将进入 NS 失效区域，除非增加迭代步数或优化系数。由于 NS 计算代价高，本文提出的缩放规律为从业者提供了基于层感知的最小必要配置方案，在不牺牲更新质量的前提下避免不必要的计算。
原文摘要 · Abstract (English)
Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon. To keep these updates tractable, Muon performs the orthonormalization with the Newton--Schulz (NS) iteration. Since NS is only approximate, directions with small singular values fail to be orthonormalized. In Muon, NS is applied to the momentum matrix at every step, yet little is known about how the singular value spectrum of these momentum matrices behaves during training, or how that behavior changes with model size. We present the first systematic study of this question. Tracking singular value quantiles of the momentum buffer across layers in models ranging from 77M to 2.8B parameters, we observe a consistent picture: after a short burn-in, the quantiles stabilize at a value determined by the layer type and model size. These stabilization values follow remarkably clean power laws in model size, with layer-dependent exponents. Layers up to mid-late depth scale very mildly with model size $M$ (around $M^{-0.25}$), so the standard 5-step NS configuration used at academic scale will continue to orthonormalize them at much larger scales. Some of the late layers, however, scale much more aggressively (up to $M^{-0.96}$) and will fall into the NS failure regime at frontier scale unless one uses more NS iterations or better-tuned coefficients. NS iterations are computationally expensive at scale; our laws give practitioners a principled, layer-aware recipe for choosing the minimum NS configuration that still orthonormalizes the directions that matter -- avoiding unnecessary computation without sacrificing update quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。