让每步推理都与最终答案挂钩,解决奖励分配模糊问题。
Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
- 将每步推理的奖励与后续步骤和最终答案关联,捕捉因果关系。
- 在多种推理策略下性能超越现有模型,提升稳定性和抗作弊能力。
- 适合需要可靠推理评估的场景,如复杂任务规划与自动化决策。
过程奖励模型(PRM)通过引导大语言模型(LLM)逐步推理以提升其推理能力,但现有方法要么孤立看待每一步推理,忽视步骤间依赖关系,要么难以将过程奖励与最终结果对齐,导致奖励信号违背推理的时间因果性,并引发信用分配模糊问题。这使得下游模型易受奖励劫持影响,性能不佳。本文提出条件奖励建模(CRM),将LLM推理视为一个通向正确答案的时序过程。每个推理步骤的奖励不仅依赖于先前步骤,还显式关联最终推理轨迹的结果。通过施加条件概率约束,该设计捕捉了推理步骤间的因果关系,并通过与最终结果的链接实现对中间步骤的精确归因,从而解决信用分配难题。此外,这种一致的概率建模使CRM产生的奖励具备更可靠的跨样本可比性。在Best-of-N采样、束搜索和强化学习等多种设置下的实验表明,CRM持续优于现有奖励模型,提供了一个增强LLM推理能力的原则性框架。尤其在无需基于真实答案的可验证奖励情况下,仍能保持鲁棒性并带来稳定性能提升。
原文摘要 · Abstract (English)
Process Reward Models (PRMs) have emerged as a promising approach to enhance the reasoning capabilities of large language models (LLMs) by guiding their step-by-step reasoning toward a final answer. However, existing PRMs either treat each reasoning step in isolation, failing to capture inter-step dependencies, or struggle to align process rewards with the final outcome. Consequently, the reward signal fails to respect temporal causality in sequential reasoning and faces ambiguous credit assignment. These limitations make downstream models vulnerable to reward hacking and lead to suboptimal performance. In this work, we propose Conditional Reward Modeling (CRM) that frames LLM reasoning as a temporal process leading to a correct answer. The reward of each reasoning step is not only conditioned on the preceding steps but also explicitly linked to the final outcome of the reasoning trajectory. By enforcing conditional probability rules, our design captures the causal relationships among reasoning steps, with the link to the outcome allowing precise attribution of each intermediate step, thereby resolving credit assignment ambiguity. Further, through this consistent probabilistic modeling, the rewards produced by CRM enable more reliable cross-sample comparison. Experiments across Best-of-N sampling, beam search and reinforcement learning demonstrate that CRM consistently outperforms existing reward models, offering a principled framework for enhancing LLM reasoning. In particular, CRM is more robust to reward hacking and delivers stable downstream improvements without relying on verifiable rewards derived from ground truth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。