通过在线验证筛选高价值提示,提升大模型强化学习训练效率
Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model
- 基于历史奖励轨迹和提示熵动态筛选高潜力提示
- 在多个数学推理任务中减少70%以上滚开展开次数
- 适合需要高效训练推理模型的研究者和工程师
强化学习(RL)已成为大语言模型在推理任务后训练的关键方法。尽管增加滚展开次数可稳定训练并提升性能,但计算开销巨大。以GRPO为例,每条提示多次滚开展开带来高昂成本,因大部分提示提供的梯度微弱,实用价值低。我们研究发现,样本效用分布不均且动态变化:最强学习信号集中于‘学习边缘’——即中等难度与高不确定性交汇处,该区域随训练进程移动。为此,我们提出HIVE(历史感知且在线验证的提示选择)框架,采用双阶段策略:先利用历史奖励轨迹进行粗筛,再以提示熵为实时代理剔除效用过时的实例。在多个数学推理基准和模型上评估显示,HIVE显著提升滚展开效率,且性能无损。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become essential for post-training large language models (LLMs) in reasoning tasks. While scaling rollouts can stabilize training and enhance performance, the computational overhead is a critical issue. In algorithms like GRPO, multiple rollouts per prompt incur prohibitive costs, as a large portion of prompts provide negligible gradients and are thus of low utility. To address this problem, we investigate how to select high-utility prompts before the rollout phase. Our experimental analysis reveals that sample utility is non-uniform and evolving: the strongest learning signals concentrate at the ``learning edge", the intersection of intermediate difficulty and high uncertainty, which shifts as training proceeds. Motivated by this, we propose HIVE (History-Informed and online-VErified prompt selection), a dual-stage framework for data-efficient RL. HIVE utilizes historical reward trajectories for coarse selection and employs prompt entropy as a real-time proxy to prune instances with stale utility. By evaluating HIVE across multiple math reasoning benchmarks and models, we show that HIVE yields significant rollout efficiency without compromising performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。