提出AR3PO算法，提升强化学习推理的采样效率与效果
Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
- 动态分配采样数量，难问题多生成，易问题少生成
- 复用正确回答，降低4.2倍采样成本，性能不降
- 适合大模型推理训练，尤其对32B模型更高效
大语言模型在推理任务中表现优异，强化学习结合可验证奖励（RLVR）已成为后训练的标准范式。代表性算法组相对策略优化（GRPO）通过归一化响应组内的奖励计算优势，但在所有响应获得相同奖励时面临优势消失问题。为此，我们提出自适应滚动与响应复用策略优化（AR3PO），引入两项新技术：自适应滚动，动态分配更多响应给困难提示，节省简单提示的计算；响应复用，利用先前生成的正确响应提供有效训练信号。我们在多个基准上对比了不同基线模型下的AR3PO表现。在7B和8B模型上，AR3PO持续优于GRPO，性能达到或超过DAPO，采样成本降低最多达4.2倍；在更大的32B模型上，其性能与DAPO相当，同时保持显著更低的采样开销。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved impressive reasoning performance, with reinforcement learning with verifiable rewards (RLVR) emerging as a standard paradigm for post-training. A representative algorithm, group relative policy optimization (GRPO) (Shao et al., 2024), computes advantages by normalizing outcome rewards within response groups, but suffers from a vanishing advantage issue when all responses in a group receive identical rewards. To address this issue, we propose Adaptive Rollout and Response Reuse Policy Optimization (AR3PO), a sampling efficient RLVR algorithm that introduces two novel techniques: adaptive rollout, which dynamically allocates more responses to difficult prompts while saving computation on easier ones, and response reuse, which leverages previously generated correct responses to provide useful training signals. We compare AR3PO with strong RLVR baselines on multiple representative benchmarks using two different families of base models. Across the 7B and 8B models, AR3PO consistently outperforms GRPO and matches or surpasses DAPO (Yu et al., 2025), reducing rollout cost by up to 4.2x. On the larger 32B model, AR3PO achieves comparable performance to DAPO at similar training steps while maintaining substantially lower rollout cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。