用经验回放提升大模型强化学习训练效率，省算力还更好
Efficient RL Training for LLMs with Experience Replay
- 设计可调节的回放缓冲区，平衡数据新旧与多样性
- 减少80%以上推理计算量，性能不降反升
- 适合算力有限但追求高效训练的研究者
尽管经验回放是强化学习中的基础技术，但在大模型后训练中仍少被探索，主流观点认为高质量性能依赖于新鲜的在线数据。本文系统研究了大模型后训练中的回放缓冲区设计，将其优化视为过时数据带来的方差、样本多样性与生成高成本之间的权衡。实验表明，在生成代价较高的情况下，严格在线采样并非最优。通过合理设计回放缓冲区，可大幅降低推理计算量，且在多数情况下保持甚至提升最终模型性能，同时维持策略熵不变。
原文摘要 · Abstract (English)
While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。