用离线策略优化提升大模型推理训练效率，解决难例学习瓶颈
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
- 基于历史难题重评估动态选样本，实现经验复用
- 在数学、规划等任务上平均比GRPO提升12.5%
- 成功解决40.7%原模型反复失败的问题，适合难例强化训练
传统基于可验证奖励的在线策略强化学习（RLVR）框架存在经验浪费和奖励同质化问题，严重影响大语言模型后训练中对困难样本的学习效率。本文提出批处理自适应策略优化（BAPO），一种离线策略的RLVR框架，通过重新评估历史难例并复用高质量样本，提升数据效率，并保证策略改进的下界。大量实验表明，BAPO在数学、规划和视觉推理任务上平均比GRPO提升12.5%；关键突破在于成功解决了40.7%基线模型始终无法解决的问题。
原文摘要 · Abstract (English)
Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。