arXiv:2607.22002cs.AI2026-07被引 1

通过动态分配推理路径提升强化学习效率,显著减少无效计算。

Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning

论文配图:Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
图 1 · 摘自论文原文
  • 根据奖励方差动态分配推理路径,优先投入高不确定样本。
  • 数学任务少2.3倍推理路径达目标准确率,编码任务少1.49倍达全通过率。
  • 适合追求高效训练的LLM推理优化研究者与工程实践者。

基于可验证奖励的强化学习(RLVR)在提升大语言模型推理能力方面表现优异,其中GRPO方法尤为成功。然而,GRPO依赖大量长链式推理路径生成,训练时间随路径数量增长,且多数路径信息量低,导致计算成本高且不稳定。现有方法或扩大路径池后筛选,或利用历史信号过滤,但性能提升有限且整体速度慢。本文提出变异性引导在线路径分配(VIGOR),不固定每例路径预算,而是从每批样本少量路径开始,迭代将额外路径分配给组奖励方差最高的样本,直至总路径预算耗尽。理论证明,在RLVR下,奖励方差控制梯度幅度,并推导出VIGOR相对于GRPO的闭式加速比,其随优化轮次增加而上升,尤其在奖励方差呈帕累托分布时。实验显示,数学推理任务中VIGOR以最多2.3倍更少的路径达目标准确率;编码任务中以1.49倍更少路径达到GRPO的最终全通过率,平均测试通过率提升3.4分。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3$\times$ fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49$\times$ fewer rollouts, and improves the coding average test pass rate by 3.4 points.

强化学习推理优化大模型高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。