通过智能筛选数据提升强化学习效率，用更少数据跑出更好结果。
Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
- 分阶段优化数据选择：离线选高质量样本，线上动态过滤低探索潜力数据。
- 仅用20%数据即达全量训练1.85倍加速，关键基准上性能超越现有方法。
- 适合追求高效训练的AI研究者，尤其在资源受限场景下有显著优势。
大型推理模型近年来借助可验证奖励强化学习（RLVR）提升了推理能力，但其扩展通常依赖大量回溯计算和大规模数据集，导致训练成本高、数据效率低。为此，我们提出DEPO——一种数据高效的策略优化流程，结合离线与在线数据选择的优化策略。离线阶段，基于多样性、影响度和适中难度筛选高质量样本子集；在线训练中，引入样本级可探索性指标，动态剔除低探索潜力样本，大幅降低回溯计算开销。同时，对未充分探索样本引入重播机制，确保充分训练，提升最终收敛性能。在五个推理基准上的实验表明，DEPO在离线与在线数据选择场景中均优于现有方法。值得注意的是，仅使用20%训练数据，该方法在AIME24上实现1.85倍加速，在AIME25上实现1.66倍加速，优于在全量数据上训练的GRPO。
原文摘要 · Abstract (English)
Recent advances in large reasoning models have leveraged reinforcement learning with verifiable rewards (RLVR) to improve reasoning capabilities. However, scaling these methods typically requires extensive rollout computation and large datasets, leading to high training costs and low data efficiency. To mitigate this issue, we propose DEPO, a Data-Efficient Policy Optimization pipeline that combines optimized strategies for both offline and online data selection. In the offline phase, we curate a high-quality subset of training samples based on diversity, influence, and appropriate difficulty. During online RLVR training, we introduce a sample-level explorability metric to dynamically filter samples with low exploration potential, thereby reducing substantial rollout computational costs. Furthermore, we incorporate a replay mechanism for under-explored samples to ensure adequate training, which enhances the model's final convergence performance. Experiments across five reasoning benchmarks show that DEPO consistently outperforms existing methods in both offline and online data selection scenarios. Notably, using only 20% of the training data, our approach achieves a 1.85 times speed-up on AIME24 and a 1.66 times speed-up on AIME25 compared to GRPO trained on the full dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。