通过隐式对比提升大模型推理的梯度稳定性。
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
- 用多轨迹对比生成每步的隐式奖励信号
- 在数学与代码推理任务上显著提升性能上限
- 适合研究强化学习与大模型推理的学者
基于大语言模型（LLMs）的多步推理强化学习通常依赖稀疏的最终奖励，导致信用分配问题严重：最终反馈被均匀传递至所有中间决策。这引发梯度方差高、训练不稳定及大量无效更新，限制模型持续优化。本文提出一种反事实对比框架，对每个输入采样多个推理轨迹，将轨迹差异视为替代决策的隐式近似，从而生成步骤敏感的隐式过程级优势估计量，将稀疏终端奖励转化为细粒度学习信号。基于此框架，我们提出隐式行为策略优化（IBPO），显著提升训练稳定性和数学与代码推理基准上的性能上限。结果表明该方向有望充分释放大模型的推理潜力。
原文摘要 · Abstract (English)
Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment problem: the final feedback is propagated uniformly across all intermediate decisions. This leads to high gradient variance, unstable training, and many ineffective updates, ultimately limiting sustained model improvement. We propose a counterfactual-comparison framework for credit assignment. For each input, the framework samples multiple reasoning trajectories and treats their differences as implicit approximations to alternative decisions. This yields an implicit process-level advantage estimator that converts sparse terminal rewards into step-sensitive learning signals. Building on this framework, we introduce Implicit Behavior Policy Optimization (IBPO), which substantially improves training stability and the performance ceiling on mathematical and code-reasoning benchmarks. Our results point to a promising direction for unlocking the reasoning potential of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。