无需预采样即可预测难易度,提升强化学习冷启动效率
ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR
- 用外部验证器构建零采样难易度先验,避免初始盲目采样
- 实验显示早期无效组减少超50%,30步前浪费采样降低近20%
- 适合需要快速收敛的高成本强化学习场景,如数学推理任务
在基于群体相对策略优化(GRPO)的可验证奖励强化学习(RLVR)中,KL-free奖励优势项依赖组内奖励方差。若组内所有轨迹均正确或均错误,则其相对优势为零,导致无梯度更新,这类沉默组占总采样量的39%。传统基于历史的提示选择需先消耗目标策略采样估计难度,造成冷启动浪费。ThinkPrior则通过一次离线验证器采样构建零采样难度先验,利用验证通过率初始化贝塔后验分布,依据预期可学性选择提示,并在训练中动态更新,不改变损失函数或优化器。在Qwen2.5-Math-7B上,16个随机种子测试显示,ThinkPrior将早期沉默组数量减半,30步前采样浪费下降近20%,最终准确率无差异。在250个提示池中,固定预算下实现资源重分配而非净节省。实际组合(ThinkPrior+DAPO)使生成采样总量减少10.6%,两阶段仍保持3840次采样更新预算。
原文摘要 · Abstract (English)
In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation. If all rollouts in a group are correct or all are wrong, their group-relative advantages are identically zero; these zero-advantage silent groups provide no reward-advantage gradient, yet uniform sampling spends 39% of a run's rollouts on them. History-based prompt selection must first spend target-policy rollouts to estimate difficulty, creating a cold start with rollout waste; ThinkPrior instead uses an external anchor in one offline pass to construct a zero-rollout difficulty prior before the first target-policy rollout. The verifier-scored anchor pass rate supplies an external-anchor initialization for a Beta posterior; ThinkPrior selects by expected learnability and then updates from training outcomes, changing neither the loss nor the optimizer. On Qwen2.5-Math-7B across sixteen seeds, ThinkPrior more than halves early silent groups and cuts wasted rollouts through step 30 by nearly a fifth, while we detect no difference in final accuracy. On this 250-prompt pool the fixed-budget result is a reallocation rather than a net saving. The measured ThinkPrior+DAPO composition reduces generated rollouts by 10.6% while both arms retain the same 3840-rollout update budget. The prior requires no target-policy rollout before the first selection, but the posterior thereafter uses target-policy outcomes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。