用反事实路径降低大模型推理中的信用分配方差，提升训练稳定性和性能上限。
Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
- 通过采样多条推理路径，用差异模拟替代决策，生成细粒度学习信号。
- 在数学与代码推理任务上，训练更稳定，性能显著优于基线方法。
- 适合研究大模型推理优化与强化学习的开发者参考。
基于大语言模型（LLMs）的多步推理强化学习通常依赖稀疏的最终奖励，导致信用分配问题严重：最终反馈被均匀传播至所有中间决策。这引发高梯度方差、训练不稳定及大量无效更新，限制了模型持续改进。本文提出一种反事实比较框架，对每个输入采样多条推理轨迹，将其差异视为替代决策的隐式近似，从而生成过程级的优势估计量，将稀疏终端奖励转化为对每一步敏感的学习信号。在此基础上，提出隐式行为策略优化（IBPO），显著提升训练稳定性与数学及代码推理基准上的性能上限。结果表明该方向有望释放LLMs的推理潜力。
原文摘要 · Abstract (English)
Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment problem: the final feedback is propagated uniformly across all intermediate decisions. This leads to high gradient variance, unstable training, and many ineffective updates, ultimately limiting sustained model improvement. We propose a counterfactual-comparison framework for credit assignment. For each input, the framework samples multiple reasoning trajectories and treats their differences as implicit approximations to alternative decisions. This yields an implicit process-level advantage estimator that converts sparse terminal rewards into step-sensitive learning signals. Building on this framework, we introduce Implicit Behavior Policy Optimization (IBPO), which substantially improves training stability and the performance ceiling on mathematical and code-reasoning benchmarks. Our results point to a promising direction for unlocking the reasoning potential of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。