arXiv:2605.05802cs.LG2026-05

通过提前终止无信息的多样本轨迹,显著降低大模型强化学习训练成本。

Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL

  • 基于部分动作序列的相似度判断,动态提前终止无价值的多样本轨迹。
  • 实验显示训练速度提升10.7%,未见任务成功率提高2.5个百分点。
  • 适合追求高效训练的大模型代理强化学习研究者使用。

群体相对强化学习(GRPO)在每个训练提示下并行采样少量轨迹,利用组内奖励差异计算每条轨迹的优势。在智能体环境中,每步对应一次大模型调用,多样本机制成为主要开销。当一组轨迹最终奖励相同时,组内方差为零,无法提供梯度信息,此类情况在实践中常见(约40%),造成大量无效计算。现有方法在提示层面过滤,但缺乏进行中轨迹的信息。本文提出一种一参数门控机制:若中间步骤的路径动作前缀平均两两编辑距离低于阈值,说明轨迹已收敛,可提前终止。在ALFWorld上使用Qwen2.5-7B运行60轮在线GRPO,四组随机种子平均结果表明,该方法使训练墙钟时间加速10.7%(置信区间不包含0),且未见任务成功率提升2.5个百分点,归因于零优势梯度批次稀释的减少。代码已公开于https://github.com/zhiyuanZhai20/selective-rollout。

原文摘要 · Abstract (English)

Group-relative RL training (GRPO) samples a small group of parallel rollouts for every training prompt and uses their within-group reward spread to compute per-trajectory advantages. In agentic environments each rollout is a long multi-turn dialogue with one LLM call per step, so this multi-sample multiplier dominates the total training cost. When every rollout of a prompt ends with the same reward, the group has zero reward variance and contributes no gradient, so the extra rollouts add no information; such groups are common in practice (typically around 40% of all groups), so the wasted-compute fraction is substantial rather than marginal. Existing methods filter such groups at the prompt level, either after their rollouts are paid for or before any rollout begins, but both decide without using information that becomes available during the rollout itself. We instead ask whether the in-group divergence between the partial trajectories at an intermediate step can already predict that the group will be zero-variance: when the parallel rollouts have already converged on the same action prefix, the group is on track to produce a single reward, and we can stop early. We propose a one-parameter gate that stops a group when the mean pairwise prefix edit distance between its partial action sequences falls below a threshold. On a 60-iteration on-policy GRPO run on ALFWorld with Qwen2.5-7B, averaged over four random seeds, the gated arm finishes 10.7% faster in wall-clock (bootstrap 95% CI excludes 0) and shifts held-out success rate on 50 unseen tasks by +2.5 pp, with the held-out gain tracing to a measurable reduction in zero-advantage gradient-batch dilution. Code is available at https://github.com/zhiyuanZhai20/selective-rollout.

强化学习大模型效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。