Muon优化器在下游任务中会放大噪声,新方法Pion通过高通滤波提升稳定性与性能。
Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR

- 用两阶段高通NS迭代替代均匀谱归一化,抑制噪声尾部分量
- 在LIBERO数据集上1500步达100%成功率,显著优于Muon和AdamW
- 适配多头注意力结构,可无额外开销保留预训练模型差异性
Muon是一种基于牛顿-舒尔兹迭代的矩阵感知优化器,通过将动量矩阵所有奇异值推向1来实现谱梯度正交化,在大语言模型预训练中表现优异。然而我们发现,其在跨模态视觉-语言-动作(VLA)训练与可验证奖励强化学习(RLVR)中存在根本缺陷:前者因动作模块梯度低秩而放大噪声尾部方向,后者因信噪比低且需保持各注意力头专属性导致归一化不稳。为此提出Pion,作为Muon的即插即用替代品,采用两阶段促进-抑制机制(高通NS迭代),使主奇异值锚定于1,同时将噪声尾部压缩至0,支持可调滤波强度。为保留预训练各头异质性,引入无需额外开销的逐头模式。在LIBERO及LIBERO-Plus数据集上,使用VLA-Adapter与VLANeXt架构,Pion均持续领先,例如在对象操作任务中仅用1,500步即达100%成功率,而Muon为97.0%,AdamW仅32.2%。该优势延伸至真实Franka Research 3机器人在DROID设置下的三类抓取放置任务。在Qwen3-1.7B/4B模型的GRPO与GMPO后训练中,Pion亦优于AdamW,在MATH与GSM8K任务上表现良好,而Muon崩溃至零。
原文摘要 · Abstract (English)
Muon is a matrix-aware optimizer that leverages Newton-Schulz (NS) iterations to enforce spectral gradient orthogonalization by driving all singular values of the momentum matrix toward 1. While this uniform spectral whitening enhances exploration and outperforms AdamW in LLM pretraining, we show it could lead to fundamental limitations beyond pretraining in two regimes: (i) cross-modality vision-language-action (VLA) training, where inherently low-rank action-module gradients cause amplification of noisy tail directions, and (ii) reinforcement learning with verifiable rewards (RLVR), where low-SNR gradients and the need to preserve per-head specialization from prior training make whitening unstable. To address these challenges, we propose Pion, a drop-in replacement for Muon that preserves its computational efficiency while replacing uniform spectral whitening with a two-stage Promotion+Suppression mechanism, which we call the high-pass NS iteration. This design induces a sharp spectral high-pass effect, anchoring dominant singular values at 1 while suppressing noisy tail components toward 0, with controllable filter strength. To preserve pretrained per-head heterogeneity, Pion also supports a per-head mode that applies updates independently across attention heads via a simple reshape, at no extra cost. In VLA training on LIBERO and LIBERO-Plus, Pion consistently outperforms both baselines across l_1-regression (VLA-Adapter) and flow-matching (VLANeXt) architectures, e.g., reaching 100% success rate on LIBERO Object after 1,500 training steps with VLA-Adapter, vs. 97.0% for Muon and only 32.2% for AdamW. The advantage of Pion further extends to a real Franka Research 3 robot with a pi_0.5 backbone under the DROID setup on three grasp-and-place tasks. In RLVR post-training on Qwen3-1.7B/4B with GRPO and GMPO, Pion also outperforms AdamW on MATH and GSM8K while Muon collapses to zero.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。