Muon在强化学习中比AdamW更稳定,可支持更大有效步长。
When Does Muon Help Agentic Reinforcement Learning?
- 采用共享KL与裁剪策略,对比Muon与AdamW在不同学习率下的表现。
- 在3B模型上,高率Muon使后期成功率提升,其隐层更新均方根达AdamW的3.53倍。
- 性能优势源于优化器的尺度惯例,而非普适排名,适合大模型强化学习调优。
Muon在大规模预训练中表现不逊于AdamW,但在强化学习微调中的适用范围尚不明确。本研究在ALFWorld这一稀疏奖励的智能体基准上,使用Qwen2.5系列0.5B至3B模型,通过三种分组目标进行评估。在共享KL和裁剪策略下,通过匹配优化器比较与AdamW学习率控制,确定了可用步长范围。AdamW对学习率响应非单调,而fan-in Muon在更激进的有效步长下仍保持稳定:在$3 \times 10^{-5}$时,经率-指标测试校正后,其后期成功率优于基线AdamW $10^{-6}$。其归一化AUC效应方向为正但不够一致;启发式匹配低率效果不稳,调优后的AdamW在3B GraphGPO上几乎追平高率Muon。高率Muon的隐藏矩阵更新均方根是AdamW的$3.53 \times$;全预算均方根匹配控制移除了后期成功率优势。结果表明,在共享KL与裁剪条件下,fan-in Muon可在优化余量充足时支持更激进且稳定的有效步长,优势在优化空间收缩、接近饱和或经AdamW调优后最为显著。规模匹配控制将此谱特性归因于Muon的尺度约定,而非建立通用优化器排序。代码已公开于https://github.com/x66ccff/verl-muon。
原文摘要 · Abstract (English)
Muon is competitive with AdamW in large-scale pre-training, but its operating regime in reinforcement-learning post-training remains unclear. We map this regime on ALFWorld, a sparse-reward agentic benchmark, using three group-based objectives and Qwen2.5 models from 0.5B to 3B. Under a shared KL and clipping recipe, matched optimizer comparisons and AdamW rate controls trace the usable step-size range. AdamW responds non-monotonically to rate, whereas fan-in Muon remains stable at a more aggressive effective step: at $3 \times 10^{-5}$ it improves late success over an AdamW $10^{-6}$ baseline after correction across rate-metric tests. Its normalized-AUC effect is directionally positive but less uniform; the heuristic-matched lower-rate effect is less consistent, and tuned AdamW nearly matches high-rate Muon at 3B GraphGPO. High-rate Muon applies $3.53 \times$ AdamW's hidden-matrix update RMS; a full-budget RMS-matched control removes the late-success gain. Together, these results identify a recipe-level operating regime in which fan-in Muon supports a more aggressive stable effective step under shared KL and clipping: the margin is largest when optimization headroom remains and contracts near saturation, after AdamW tuning, or under magnitude matching. The scale-matched control ties this spectral effect to Muon's scale convention rather than establishing a universal optimizer ranking. Code is available at https://github.com/x66ccff/verl-muon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。