让大模型通过离策略强化学习高效提升推理能力
LLMs Can Learn to Reason Via Off-Policy RL
- 提出新算法OAPL，直接利用离策略数据训练，无需修正策略差异
- 数学竞赛和编码任务表现优于现有方法，训练生成量减少3倍
- 支持超大延迟（400步以上）的训练-推理差异，适合真实部署场景
大型语言模型的强化学习常采用在线策略算法（如PPO或GRPO），但分布式训练导致策略滞后，且训练与推理策略不一致，使数据天然为离策略。现有方法试图通过重要性采样或修改推理引擎使其更像在线策略。本文提出新算法OAPL——基于最优优势的离策略策略优化，主动接纳离策略特性，无需额外修改。实验表明，OAPL在数学竞赛基准上优于带重要性采样的GRPO，编码任务上媲美公开模型DeepCoder，训练生成次数减少3倍。此外，模型在测试时的Pass@k指标具有更好扩展性。该方法可处理超过400梯度步的训练-推理延迟，比以往方法离策略程度高100倍。
原文摘要 · Abstract (English)
Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。