针对大模型推理训练效率低的问题，提出按回滚样本优先重播机制。
Rollout-Level Advantage-Prioritized Experience Replay for GRPO
- 按回滚样本而非整组存储与采样，结合年龄淘汰机制控滞留
- 通过优势值大小优先重播，大优势回滚可重复利用，提升样本价值
- 在多个数学基准上优于基线，尤其40亿模型规模下提升最显著
基于可验证奖励的GRPO是后训练推理大模型的标准方法，但样本效率低。每个回滚仅用于一次梯度更新即丢弃。直接重播不适用，因大模型策略每步更新后快速漂移，导致存储回滚迅速过时并破坏训练。本文提出一种回滚级重播缓冲区，存储并采样单个回滚而非整组。缓冲区通过年龄淘汰控制滞留：超过tau_max训练步数的回滚将被移除。同时通过新鲜锚定组合保留在线数据：每批保持新鲜在线回滚，并额外拼接从缓冲区独立抽取的重播回滚。按回滚级优势幅度优先重播，对高优势回滚进行循环利用。在三个Qwen3-Base模型规模（1.8B、7B、4B）上，五项数学基准测试中，本方法均优于GRPO和朴素重播基线。增益在各规模均为正，且随模型增大而上升。最大增益为4B模型在五基准平均上+4.35个百分点。在联合衡量准确率与令牌效率的AES指标下，4B模型的效率优势最大，达+0.579。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs. It remains sample inefficient. Each rollout is used for a single gradient update and then discarded. Naive replay is not well suited in this setting because LLM policies drift quickly per gradient step. Stored rollouts therefore become stale and can destabilize training. We propose a rollout-level replay buffer for GRPO that stores and samples individual rollouts rather than whole groups. The buffer bounds staleness through age eviction. Any rollout older than tau_max training steps is removed. The buffer also preserves on-policy data via fresh-anchored composition. Each batch keeps its fresh on-policy rollouts and then concatenates replay rollouts drawn separately from the buffer. We prioritize replay by per-rollout advantage magnitude and recycle individual rollouts whose advantages are large. Across three Qwen3-Base scales on five math benchmarks, our method outperforms GRPO and naive replay baselines. Gains are positive at every scale and grow with model size. The largest gain is +4.35 pp on the five-benchmark average at 4B. Under an AES metric that jointly measures accuracy and token efficiency, the efficiency margin over GRPO is again largest at 4B, at +0.579.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。