通过跳过无效提示，让大模型推理强化学习更快更省资源。
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
- 训练时先预测并跳过无价值提示，减少无效计算。
- 在多个数学推理模型上实现最高2.4倍的采样速度提升。
- 适合追求高效训练的大模型研究者与部署团队。
强化学习（如PPO和GRPO）推动了大语言模型推理能力的突破。扩大采样范围可筛选高质量数据用于训练，提升稳定性与性能，但带来显著计算开销。本文分析发现提示价值具有强时间一致性：一个训练周期中无价值的提示，在后续周期中仍可能无效。基于此，我们提出GRESO（GRPO with Efficient Selective Rollout），一种轻量级在线预采样过滤算法，利用奖励训练动态预测并跳过无信息提示。在Qwen2.5-Math-1.5B、DeepSeek-R1-Distill-Qwen-1.5B和Qwen2.5-Math-7B等多个数学推理模型与基准测试上验证，GRESO实现了最高2.4倍的采样耗时加速，总训练时间提速达2.0倍，且无准确率下降。
原文摘要 · Abstract (English)
Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance. However, this comes at the cost of significant computational overhead. In this paper, we show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, such as Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, we show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。