arXiv:2607.26253cs.LG2026-07

用早期反馈动态分配计算预算,显著减少强化学习中的无效推理开销。

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

论文配图:Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
图 1 · 摘自论文原文
  • 基于贝叶斯更新与序贯停止策略,早判有效/无效提示并释放资源。
  • 在1.5B/3B模型上比基线少22%推理量,精度接近最优水平。
  • 适合算力有限但需高效训练的RLVR研究者,尤其数学推理与规划任务。

基于可验证奖励的强化学习(RLVR)受限于推理生成成本,许多采样提示形成全对或全错的饱和组,其零奖励方差无法提供策略梯度信号。现有方法或过度采样后丢弃饱和组(动态采样),增加额外推理开销;或采样前预测提示难度,但在策略变化时易失效。我们发现一组推理的有效性通常在前几次推理中即可确定,因此对已判定提示投入完整组推理是浪费。我们将每步推理收集建模为预算约束下的序贯分配(最优停止)问题,提出SARA(Sequential Adaptive Rollout Allocation)。SARA维护每个提示成功概率的贝塔后验,评估闭式组有效性预测器,并采用双阈值、SPRT风格规则:及时确认有效组,短探后放弃饱和组,将释放预算重用于新提示,无需额外预测推理。我们证明了弃用可靠性、预期推理节省、固定预算下产出优势,以及有效组产出与GRPO梯度范数的关联。在单张GPU上使用1.5B/3B模型进行数学推理与规划任务,SARA在低于DS基准的情况下匹配其性能,且推理量减少22%;将SARA与DPS结合,达到最高精度,略超DS,但推理量仅需67%(近似均匀成本)。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).

强化学习推理优化预算分配高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。