让大模型强化学习更高效，用旧数据反复训练
Off-Policy Value-Based Reinforcement Learning for Large Language Models
- 基于贝尔曼更新结合内部一致性与结果验证信号
- 在数学推理任务上收敛更快，性能优于GRPO
- 适合追求高效训练的大模型研究者
提升数据利用效率对长序列任务的强化学习至关重要，因为生成轨迹成本高昂。当前主流的大模型强化学习方法多为在线策略：每批数据仅使用一次，用完即丢，导致样本效率低下。本文提出一种新的基于价值的离线策略强化学习框架ReVal，通过结合逐步内部一致性信号和轨迹级结果验证信号，实现自然的离线学习。ReVal支持基于回放缓冲区的训练，可高效复用历史轨迹。在标准数学推理基准测试中，ReVal不仅收敛速度更快，最终性能也优于GRPO。在DeepSeek-R1-Distill-1.5B模型上，相较于GRPO，ReVal在AIME24上提升2.7%，在跨域基准GPQA上提升4.5%。结果表明，基于价值的强化学习是大模型训练中可行且高效的替代方案。
原文摘要 · Abstract (English)
Improving data utilization efficiency is critical for scaling reinforcement learning (RL) for long-horizon tasks where generating trajectories is expensive. However, the dominant RL methods for LLMs are largely on-policy: they update each batch of data only once, discard it, and then collect fresh samples, resulting in poor sample efficiency. In this work, we explore an alternative value-based RL framework for LLMs that naturally enables off-policy learning. We propose ReVal, a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification. ReVal naturally supports replay-buffer-based training, allowing efficient reuse of past trajectories. Experiments on standard mathematical reasoning benchmarks show that ReVal not only converges faster but also outperforms GRPO in final performance. On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO. These results suggest that value-based RL is a practical alternative to policy-based methods for LLM training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。