arXiv:2605.07114cs.LG2026-05被引 1

根据提示的纠错潜力动态分配采样次数,提升大模型推理效率。

Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR

论文配图:Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR
图 1 · 摘自论文原文
  • 按提示的纠错潜力动态分配采样次数,而非固定分配。
  • 在12组配置中10组表现优于基准方法,且计算量相当。
  • 无需训练即可兼容现有方法,适合追求高效推理的研究者。

基于可验证奖励的强化学习(RLVR)已成为提升大语言模型推理能力的核心范式。现有的分组策略(如GRPO)对每个提示均分配固定数量的采样路径,导致计算资源分配不均:某些提示因已有足够样本而冗余采样,另一些提示则缺乏足够探索以发现正确路径。为此,本文提出“命中效用”(hit utility),即在增加采样后至少有一条路径正确的后验概率。基于此,我们设计了无需学习的最优采样分配策略HORA,其在每批分配中最大化总命中效用。HORA自适应重分配采样预算,同时保持下游奖励评估与分组优势估计不变。在四个数学推理基准和三种模型规模下,相较于计算量匹配的GRPO,HORA在十二组配置中有十组实现更高Pass@K,仅一组持平,一组达到饱和。该方法还可无缝兼容其他分组估计器(如RLOO)。消融实验表明,HORA使用的均匀先验在性能上已媲美五种提示条件下的学习型先验。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) has emerged as a central paradigm for improving the reasoning capabilities of large language models. Group-based policy optimization methods, such as GRPO, typically allocate a fixed number of rollouts to every prompt. This uniform allocation can be inefficient: it over-allocates compute to prompts whose sampled groups are already saturated while under-exploring prompts for which additional samples may reveal useful correct trajectories. To address this limitation, we introduce hit utility, the posterior probability that at least one rollout in a proposed additional allocation for a prompt will be correct. Building on this notion, we propose Hit-Utility Optimal Rollout Allocation (HORA), a learning-free rollout allocation policy that maximizes total posterior hit utility within each allocation batch. HORA adaptively reallocates rollout budgets while leaving the downstream reward evaluation and group-based advantage estimator unchanged. Across four mathematical reasoning benchmarks and three model scales, HORA preserves comparable Pass@1 and improves Pass@K over compute-matched GRPO in ten of twelve model--benchmark configurations, with one tie and one saturated exception. It is also drop-in compatible with other group-based estimators such as RLOO. Ablation studies indicate that the uniform prior used by HORA is competitive with five prompt-conditioned learned-prior alternatives.

强化学习大模型推理采样优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。