改进优化算法，让模型训练更快更准。
Beyond the Matrix Sign: Quadratic Spectral Descent
- 用二次模型替代线性模型，更准确捕捉梯度曲率。
- 在相同条件下，验证损失更低，训练速度提升最多8.49%。
- 适合追求高效训练的深度学习研究者与工程师。
Muon可被理解为在谱范数球内优化一个线性局部目标，其更新保持梯度的奇异方向，并为所有活跃奇异模态分配相同幅度。我们探究当引入局部曲率时，这两项性质是否仍最优。为此，保留Muon的谱范数约束，将线性局部模型替换为二次模型，提出新方法 extit{Quadratic Spectral Descent}（QSD）。研究表明，曲率可同时改变最优更新的奇异值和奇异方向。为使QSD实用，采用克罗内克分解统计量近似曲率，并通过少量Frank--Wolfe迭代求解约束二次问题，每次迭代包含闭式解的矩阵符号子问题。此外，提供最优性证明、与Muon在相同二次代理下的比较，以及内层求解器$O(1/K)$的收敛速率。在GPT预训练实验中，QSD始终优于Muon及其近期变体，在相同验证损失下，墙钟训练时间减少最多达8.49%。
原文摘要 · Abstract (English)
Muon can be interpreted as optimizing a linear local objective over a spectral-norm ball. This gives a matrix-sign update that preserves the singular directions of the gradient and assigns the same magnitude to all active singular modes. We ask whether these two properties remain optimal when local curvature is taken into account. To answer this question, we keep Muon's spectral-norm constraint unchanged and replace the linear local model with a quadratic one. We call the resulting method \emph{Quadratic Spectral Descent} (QSD). We show that curvature can change both the singular values and the singular directions of the optimal update. To make QSD practical, we approximate curvature with Kronecker-factored statistics and solve the constrained quadratic with a small number of Frank--Wolfe steps, each of which has a closed-form matrix-sign subproblem. We further provide an optimality certificate, a comparison with Muon under the same quadratic surrogate, and an $O(1/K)$ convergence rate for the inner solver. Experiments on GPT pre-training show that QSD consistently improves validation loss over Muon and recent Muon variants, and reduces wall-clock training time by up to $8.49\%$ at matched validation loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。