用回放机制提升大模型强化学习效率，性能显著优于现有方法。
RePO: Replay-Enhanced Policy Optimization
- 通过回放缓冲区获取多样化的离策略样本，扩展优化数据集。
- 在7个数学推理任务上，性能比GRPO提升18.4和4.1分。
- 提升48%有效优化步数，仅增加15%计算开销，适合高效训练需求。
强化学习对优化大语言模型至关重要。近期的组相对策略优化（GRPO）通过每提示生成多个在线策略输出来估计优势，导致高计算成本和低数据效率。为此，我们提出回放增强策略优化（RePO），利用多种回放策略从回放缓冲区中检索离策略样本，使每个提示的策略优化基于更广泛、更多样的样本。在五个大语言模型上，七个数学推理基准上的实验表明，与GRPO相比，RePO在Qwen2.5-Math-1.5B和Qwen3-1.7B上分别获得18.4和4.1分的绝对平均性能提升。进一步分析显示，对于Qwen3-1.7B，RePO在计算成本增加15%的同时，将有效优化步骤数提高48%，且在线策略与离策略样本数均设为8。代码仓库地址：https://github.com/SihengLi99/RePO。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is vital for optimizing large language models (LLMs). Recent Group Relative Policy Optimization (GRPO) estimates advantages using multiple on-policy outputs per prompt, leading to high computational costs and low data efficiency. To address this, we introduce Replay-Enhanced Policy Optimization (RePO), which leverages diverse replay strategies to retrieve off-policy samples from a replay buffer, allowing policy optimization based on a broader and more diverse set of samples for each prompt. Experiments on five LLMs across seven mathematical reasoning benchmarks demonstrate that RePO achieves absolute average performance gains of $18.4$ and $4.1$ points for Qwen2.5-Math-1.5B and Qwen3-1.7B, respectively, compared to GRPO. Further analysis indicates that RePO increases computational cost by $15\%$ while raising the number of effective optimization steps by $48\%$ for Qwen3-1.7B, with both on-policy and off-policy sample numbers set to $8$. The repository can be accessed at https://github.com/SihengLi99/RePO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。