用回放高质推理路径提升大模型强化学习效率与准确率
RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning
- 分两阶段训练，回放验证过的优质推理轨迹
- 在AIME-2024等数据集上准确率提升1.7%~5.2%
- 适合需要稳定高效微调大模型推理能力的研究者
大语言模型的强化学习训练能耗高且易不稳定，策略可能逐渐偏离预训练权重。我们提出RLEP（Reinforcement Learning with Experience Replay），一种两阶段框架：先收集经验证的推理轨迹，再在后续训练中回放这些成功路径。每次更新时，策略在新生成轨迹与回放的成功样本混合的小批量上优化。通过回放高质量示例，RLEP避免无效探索，聚焦于有前景的推理路径，实现更快收敛和更强最终性能。在Qwen2.5-Math-7B基模型上，RLEP以更少更新次数达到基线峰值准确率，并显著超越基线：AIME-2024从38.2%提升至39.9%，AIME-2025从19.8%提升至22.3%，AMC-2023从77.0%提升至82.2%。代码、数据集和检查点已公开于https://github.com/Kwai-Klear/RLEP，便于复现与进一步研究。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for large language models is an energy-intensive endeavor: training can be unstable, and the policy may gradually drift away from its pretrained weights. We present \emph{RLEP}\, -- \,Reinforcement Learning with Experience rePlay\, -- \,a two-phase framework that first collects verified trajectories and then replays them during subsequent training. At every update step, the policy is optimized on mini-batches that blend newly generated rollouts with these replayed successes. By replaying high-quality examples, RLEP steers the model away from fruitless exploration, focuses learning on promising reasoning paths, and delivers both faster convergence and stronger final performance. On the Qwen2.5-Math-7B base model, RLEP reaches baseline peak accuracy with substantially fewer updates and ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%, on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our code, datasets, and checkpoints are publicly available at https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。