根据提示成功率动态分配采样次数，提升强化学习训练效率。
Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards
- 用轻量高斯过程预测每个提示的成功概率
- 基于方差最小化分配采样预算，提升梯度更新精度
- 在多个基准上优于固定或启发式分配策略
具有可验证奖励的强化学习中，采样效率是关键瓶颈。现有基于分组的策略优化方法（如 GRPO）对所有训练提示采用固定采样次数，隐含假设所有提示信息量相同，导致计算资源浪费并阻碍训练进展。本文提出 VIP（Variance-Informed Predictive），一种基于方差感知的预测性采样分配策略，旨在最小化策略更新的期望梯度方差。每轮迭代中，VIP 使用轻量级高斯过程模型，基于近期采样结果预测各提示的成功概率，并将其转化为方差估计，输入凸优化问题以在硬性计算预算约束下确定最优采样分配。实验表明，VIP 在多个基准上均显著提升采样效率，性能优于均匀分配或启发式策略。
原文摘要 · Abstract (English)
Sampling efficiency is a key bottleneck in reinforcement learning with verifiable rewards. Existing group-based policy optimization methods, such as GRPO, allocate a fixed number of rollouts for all training prompts. This uniform allocation implicitly treats all prompts as equally informative, and could lead to inefficient computational budget usage and impede training progress. We introduce VIP, a Variance-Informed Predictive allocation strategy that allocates a given rollout budget to the prompts in the incumbent batch to minimize the expected gradient variance of the policy update. At each iteration, VIP uses a lightweight Gaussian process model to predict per-prompt success probabilities based on recent rollouts. These probability predictions are translated into variance estimates, which are then fed into a convex optimization problem to determine the optimal rollout allocations under a hard compute budget constraint. Empirical results show that VIP consistently improves sampling efficiency and achieves higher performance than uniform or heuristic allocation strategies in multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。