用动态优先级选题,让大模型强化学习后训练更高效
Prioritized Replay for RL Post-training
- 根据解题成功率自动计算优先级,聚焦中等难度问题
- 相比传统方法,训练效率提升显著,且无需预设难易等级
- 适合希望自动化优化训练数据的AI研发人员
我们提出一种面向大型语言模型强化学习后训练的问题级优先重放框架。基于深度强化学习中优先经验回放的洞察,以及发现中等成功率的轨迹在GRPO等方法下能产生更强学习信号这一现象,我们的方法通过基于实测成功率统计的简单模型驱动优先级分数来选择问题。与强调早期训练简单任务的传统课程策略不同,该方法自然将注意力集中于既非始终成功也非始终失败的问题,同时降低对梯度贡献小问题的关注。该方法实现了无需预定义难度层级、额外预测器或外部标签的持续自适应优先化过程。我们还引入轻量级部署机制,包括基于堆的优先采样和定期重测已解决与未解决的问题,以缓解饥饿与遗忘问题。整体上,该方法为手动设计课程提供了一种原理清晰且可扩展的替代方案,并使数据选择直接契合基于GRPO的后训练动态。
原文摘要 · Abstract (English)
We introduce a problem-level prioritization framework for RL post-training of large language models. Building on insights from prioritized replay in deep RL, as well as prior observations that rollouts with intermediate success rates tend to produce stronger learning signals under methods such as GRPO, our approach selects problems according to a simple, model-driven priority score derived from empirical success statistics. In contrast to conventional curriculum strategies that emphasize easier tasks early in training, the resulting schedule naturally focuses training on problems that are neither consistently solved nor consistently failed, while deprioritizing those that contribute little gradient information. The method yields a continuously adapting and automatic prioritization process that requires no predefined difficulty tiers, auxiliary predictors, or external labels. We further introduce lightweight mechanisms for practical deployment, including heap-based prioritized sampling and periodic retesting of solved and unsolved problems to mitigate starvation and forgetting. Overall, the approach offers a principled and scalable alternative to manually designed curricula while aligning data selection directly with the dynamics of GRPO-based post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。