arXiv:2512.07710cs.AIcs.LG2025-12被引 1

百亿美元级MoE模型用强化学习训练,让每个提示都有效。

Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoE

  • 通过多阶段去零方差机制,剔除无效提示,提升训练效率。
  • 引入自适应熵优化方法,稳定长序列学习动态,避免奖励偏差。
  • 适合大规模MoE模型的高效强化学习系统,支持高吞吐训练。

我们提出CompassMax-V3-Thinking,一个百亿级MoE推理模型,采用新设计的强化学习框架,核心原则是每个提示都必须有价值。规模扩展暴露了多项关键效率问题:零方差提示浪费采样、长时序下重要性采样不稳定、标准奖励模型引发优势反转,以及采样处理的系统性瓶颈。为此,我们提出四项统一创新:(1) 多阶段零方差消除,过滤非信息提示,稳定基于组的策略优化(如GRPO);(2) ESPO,一种自适应熵优化方法,平衡词级与序列级重要性采样,维持稳定学习动态;(3) 路由重播策略,对齐训练与推理时的MoE路由行为,缓解训练-推理差异,并调整奖励模型防止优势反转;(4) 高吞吐强化学习系统,支持FP8精度采样、重叠奖励计算与长度感知调度,彻底消除性能瓶颈。这些贡献构成一体化流水线,使百亿级MoE模型的强化学习训练稳定且高效。模型在内部及公开评测中均表现优异。

原文摘要 · Abstract (English)

We present CompassMax-V3-Thinking, a hundred-billion-scale MoE reasoning model trained with a new RL framework built on one principle: each prompt must matter. Scaling RL to this size exposes critical inefficiencies-zero-variance prompts that waste rollouts, unstable importance sampling over long horizons, advantage inversion from standard reward models, and systemic bottlenecks in rollout processing. To overcome these challenges, we introduce several unified innovations: (1) Multi-Stage Zero-Variance Elimination, which filters out non-informative prompts and stabilizes group-based policy optimization (e.g. GRPO) by removing wasted rollouts; (2) ESPO, an entropy-adaptive optimization method that balances token-level and sequence-level importance sampling to maintain stable learning dynamics; (3) a Router Replay strategy that aligns training-time MoE router decisions with inference-time behavior to mitigate train-infer discrepancies, coupled with a reward model adjustment to prevent advantage inversion; (4) a high-throughput RL system with FP8-precision rollouts, overlapped reward computation, and length-aware scheduling to eliminate performance bottlenecks. Together, these contributions form a cohesive pipeline that makes RL on hundred-billion-scale MoE models stable and efficient. The resulting model delivers strong performance across both internal and public evaluations.

强化学习MoE模型高效训练大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。