提出新型优化器Muon^p，用分数阶谱幂提升模型微调效果。
Muon$^p$: Muon with Fractional Spectral Powers
- 用分数阶谱幂替代梯度，介于传统优化与Muon之间。
- 在十亿参数模型上显著降低验证困惑度，提升下游任务性能。
- 适合大规模模型微调，尤其当保留奇异值信息有帮助时。
Muon是一种日益流行的优化器，通过将梯度 $G=USV^ op$ 替换为其极分解因子 $UV^ op$ 来平滑奇异谱。然而完全平滑会丢失可能影响自适应性的奇异值信息。本文提出 Muon$^p$，一种基于分数阶谱幂更新 $US^pV^ op$（$p o(0,1)$）的优化器，实现从 Muon 到梯度下降的连续过渡。为使其实用，证明了分数阶谱幂无法通过固定一元多项式迭代计算，并推导出仅用矩阵乘法即可实现的低阶奇数双变量递推公式，保持了 Muon 的纯矩阵乘法结构和计算复杂度。实验表明，Muon$^p$ 在 Schatten $q$-范数下最大化损失的线性改进，其中 $q=1+ rac{1}{p}$。在十亿级模型微调中，它显著提升了验证困惑度和下游任务表现。我们还通过奇异谱几何分析了其适用边界，揭示了在何种情况下保留奇异谱可带来显著增益，并提供了一种系统化方法。
原文摘要 · Abstract (English)
Muon is an increasingly widely used optimizer that replaces a gradient $G=USV^\top$ with its polar factor $UV^\top$, thereby flattening the singular spectrum. However, full flattening discards singular-value information that may matter for adaptation. We introduce Muon$^p$, a Muon-style optimizer that instead uses fractional spectral-power updates $US^pV^\top$ for rational $p\in(0,1)$, interpolating between Muon and gradient descent. To make it practical, we prove that fractional spectral powers cannot be computed by any fixed univariate polynomial iteration, and furthermore derive low-degree odd bivariate recurrences that approximate $US^pV^\top$ using only matrix multiplications, preserving Muon's matrix-multiplication-only structure and compute complexity. We show that Muon$^p$ maximizes the linear improvement in loss under the Schatten $q$-norm for $q=1+\frac{1}{p}$. Empirically, Muon$^p$ is especially effective for finetuning: on billion-scale models, Muon$^p$ improves validation perplexity and downstream task performance. We further analyze when Muon$^p$ is less suitable, through the lens of spectral geometry. Our results reveal important insights on when preserving the singular spectrum can bring significant gains, and introduce a principled way to achieve them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。