通过少量推理计算优化强化学习训练效率,减少无效计算浪费。
sGPO: Trading Inference FLOPs for Training Efficiency in RLVR

- 用初始策略快速评估查询难易度,动态调整训练采样组大小。
- 训练计算量减少3倍,且性能不低于基准方法。
- 适合追求高效训练的RLVR研究者和资源受限场景。
标准强化学习带可验证奖励(RLVR)训练对每个查询分配固定采样预算,不考虑查询难度对当前策略的影响,导致两类对称失效:简单查询因策略已解决而优势接近零,无法求解的查询则无信号反馈。这两种情况均浪费训练浮点运算(FLOPs)且不贡献学习梯度。本文提出排序分组策略优化(sGPO),通过少量推理计算换取大幅降低训练计算开销。核心思想是利用廉价的推理计算作为查询难度的离线代理:对每个查询在初始策略下并行生成小批量样本,获得模型感知的经验成功率。据此将训练采样组大小设为成功率的倒数,实现每轮采样的最大优势提取。一次预评估过程同时完成数据过滤(剔除简单查询、子采样难题)、自适应组大小分配与课程学习构建(从易到难调度)。sGPO在包含预评估成本的前提下,总训练计算量减少3倍,性能达到或超过基线。
原文摘要 · Abstract (English)
Standard Reinforcement Learning with Verifiable Rewards (RLVR) training allocates a fixed rollout budget to every query, without regard for what each query's difficulty means for the current policy. This leads to two symmetric failure modes: easy queries produce near-zero advantage because the policy already solves them, while unsolvable queries produce no signal because the policy never solves them. Both regimes waste training FLOPs without contributing to a learning gradient. We introduce sorted Group Policy Optimization (sGPO), a compute-efficient strategy that trades a small budget of inference FLOPs for a large reduction in wasted training FLOPs. The key insight is that cheap inference compute can serve as a single offline proxy for query difficulty. By generating a small batch of parallel samples per query under the initial policy, we obtain a model-aware empirical success rate. This motivates setting the training rollout group size to the inverse of this success rate, a practical rule that maximizes sample efficiency by extracting the most advantage per generated rollout. This single profiling pass simultaneously drives data filtering (removing trivial queries and sub-sampling unsolvable ones), adaptive group size allocation, and curriculum construction (scheduling queries from easy to hard). sGPO matches or exceeds baseline performance while reducing total training compute by a factor of three, with the upfront inference profiling cost included.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。