根据提示质量动态分配强化学习采样次数，提升训练效率
Cross-Epoch Adaptive Rollout Optimization for RL Post-Training
- 基于贝叶斯方差估计每个提示的采样价值，自适应分配采样预算
- 在多个数学推理任务上优于固定采样策略，样本效率显著提升
- 适合大规模语言模型后训练中资源有限时使用
大语言模型后训练常依赖强化学习，需对每个提示采样多次。但现有方法对所有提示采用固定采样数，忽视不同提示提供的训练信号差异。本文研究在全局固定预算下自适应采样分配，将问题建模为具有提示级递减回报的在线资源分配。提出CERO方法，为每个提示维护成功概率的贝塔后验，用后验伯努利方差作为额外采样的价值估计。据此构建凹形、饱和型效用函数，使跨提示与跨轮次的决策受全局预算耦合。由于目标函数时间不可分，推导Fenchel对偶形式，并通过投影在线梯度下降更新提示级与预算级对偶变量。在固定提示效用下，证明了相对于离线最优分配的$O(\sqrt{K})$后悔界。在数学推理任务上的实验表明，CERO在多个开源大模型和基准上持续优于GRPO，验证了自适应采样预算可提升样本效率。
原文摘要 · Abstract (English)
LLM post-training often relies on reinforcement learning methods that sample multiple rollouts per prompt, yet most existing approaches use a fixed rollout budget for every prompt, despite large differences in the training signal different prompts provide. In this paper, we study adaptive rollout allocation under a fixed global budget and formulate the problem as online resource allocation with prompt-level diminishing returns. Our method, CERO, maintains a Beta posterior over each prompt's success probability and uses the posterior expected Bernoulli variance as a Bayesian estimate of the value of additional rollouts. We use this estimate to construct a concave, saturating utility over cumulative allocations, yielding an objective in which decisions across prompts and epochs are coupled by the global budget. Since the resulting objective is temporally nonseparable, we derive a Fenchel-dual reformulation and update both prompt-level and budget-level dual variables via projected online gradient descent. Under fixed prompt utilities, we prove an $O(\sqrt{K})$ regret bound against the offline allocation benchmark. Experiments on mathematical-reasoning problems show that CERO consistently outperforms GRPO across multiple open-weight LLMs and benchmarks, demonstrating that adaptive rollout budgeting can improve sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。