通过筛选关键推理路径，加速大模型强化学习训练。
Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
- 只用精选的推理轨迹更新模型，降低计算开销。
- 在多个基准上比原方法快1.7倍以上，性能不变。
- 适合资源有限但需高效训练大模型的场景。
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型推理能力的主流方法。然而，该方法存在显著的算力与内存不对称：推理生成高度并行且内存占用低，而策略更新则通信密集且内存消耗大。为此，我们提出PODS（下采样策略优化），通过从所有推理轨迹中选择性地保留部分样本进行策略更新，实现推理生成与策略更新解耦，在保持学习质量的同时大幅降低更新成本。我们提出一种基于最大方差的下采样准则，以最大化奖励多样性，并设计了 $O(n ext{log}n)$ 的高效实现。实验表明，采用PODS的分组相对策略优化（GRPO）在不同推理基准和硬件配置下，达到原始GRPO最高测试准确率的速度至少提升1.7倍。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has emerged as the leading approach for enhancing reasoning capabilities in large language models. However, it faces a fundamental compute and memory asymmetry: rollout generation is embarrassingly parallel and memory-light, whereas policy updates are communication-heavy and memory-intensive. To address this, we introduce PODS (Policy Optimization with Down-Sampling), which decouples rollout generation from policy updates by training only on a strategically selected subset of rollouts, maintaining learning quality while dramatically reducing update costs. We propose a principled subset selection criterion, max-variance down-sampling, that maximizes reward diversity, and provide an efficient $O(n\log n)$ implementation. Empirically, Group Relative Policy Optimization (GRPO) with PODS achieves the peak test accuracy of vanilla GRPO at least $\mathbf{1.7\times}$ faster across the different reasoning benchmarks and hardware configurations we tested.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。