用过程奖励引导数学推理,避免模型被虚假流畅误导。
LLM Reasoning with Process Rewards for Outcome-Guided Steps

- 将过程奖励转为同答案组内的相对偏好,消除偏差
- 在多个数学数据集上提升准确率,且减少所需推理次数
- 无需额外训练模块,适合追求稳定可靠的推理系统
大语言模型的数学推理能力通过使用可验证奖励的强化学习显著提升,此类方法仅优化最终答案正确性,导致长链条多步解题中反馈稀疏,难以指导中间错误。近期工作引入过程奖励模型(PRM)对中间步骤评分以提供更密集监督。然而,PRM评分常与最终正确性不一致,可能奖励表面流畅但结果错误的推理路径。若直接作为绝对奖励优化,会放大这类错误模式并引发奖励黑客行为。本文提出PROGRS框架,利用PRM但以最终正确性为主导。该方法将过程奖励视为同一问题下不同解题路径间的相对偏好,采用结果条件中心化:对错误轨迹的PRM得分进行调整,使其在每个提示组内均值为零。这消除了系统性偏差,同时保留了有效的排序信息。PROGRS结合冻结的分位数回归型PRM与多尺度一致性评估器,并将修正后的过程奖励融入组相对策略优化(GRPO),无需额外辅助目标或可训练组件。在MATH-500、AMC、AIME、MinervaMath和OlympiadBench五个数据集上,PROGRS持续优于仅依赖结果的基线模型,且在更少采样轮次下实现更强性能。结果表明,结果条件中心化使得过程奖励能安全而有效地用于数学推理。
原文摘要 · Abstract (English)
Mathematical reasoning in large language models has improved substantially with reinforcement learning using verifiable rewards, where final answers can be checked automatically and converted into reliable training signals. Most such pipelines optimize outcome correctness only, which yields sparse feedback for long, multi-step solutions and offers limited guidance on intermediate reasoning errors. Recent work therefore introduces process reward models (PRMs) to score intermediate steps and provide denser supervision. In practice, PRM scores are often imperfectly aligned with final correctness and can reward locally fluent reasoning that still ends in an incorrect answer. When optimized as absolute rewards, such signals can amplify fluent failure modes and induce reward hacking. We propose PROGRS, a framework that leverages PRMs while keeping outcome correctness dominant. PROGRS treats process rewards as relative preferences within outcome groups rather than absolute targets. We introduce outcome-conditioned centering, which shifts PRM scores of incorrect trajectories to have zero mean within each prompt group. It removes systematic bias while preserving informative rankings. PROGRS combines a frozen quantile-regression PRM with a multi-scale coherence evaluator. We integrate the resulting centered process bonus into Group Relative Policy Optimization (GRPO) without auxiliary objectives or additional trainable components. Across MATH-500, AMC, AIME, MinervaMath, and OlympiadBench, PROGRS consistently improves Pass@1 over outcome-only baselines and achieves stronger performance with fewer rollouts. These results show that outcome-conditioned centering enables safe and effective use of process rewards for mathematical reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。