分离推理与训练策略，提升大模型推理准确率
R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning
- 用残差回溯头解耦训练轨迹与推理输出，避免梯度干扰
- MATH-500和APPS数据集上准确率分别提升3.4%和1.3%
- 生成更多样、更少长度偏差的推理路径，适合复杂逻辑任务
现有强化学习方法在大模型推理中隐含假设：训练轨迹生成策略与推理响应策略应一致。我们指出这一假设带来误导性归纳偏置：优化训练轨迹分布侧重信息梯度，而推理最优分布强调准确性和一致性。强制统一两者会纠缠梯度并抑制探索。为此提出R$^2$PO（残差回溯策略优化），在策略之上附加轻量级残差回溯头，实现训练轨迹与推理输出的解耦，在训练中多样化回溯路径的同时保持推理生成不变。实验表明，R$^2$PO持续优于基线，在MATH-500上平均准确率提升3.4%，APPS上提升1.3%，且回溯路径更丰富、长度偏差更小。代码已公开于https://github.com/RRPO-ARR/Code。
原文摘要 · Abstract (English)
Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R$^2$PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R$^2$PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。