arXiv:2601.18984cs.LGcs.CL2026-01ACL被引 7

通过精准定位错误起点,实现更稳定的推理强化学习。

Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning

  • 用过程奖励模型定位首个错误步骤,只惩罚错误后内容
  • 在多个基准上,准确率比传统方法提升12%-18%
  • 适合需要可解释推理的AI系统开发人员

强化学习(RL)已成为提升大语言模型(LLM)推理能力的重要框架。然而,现有大多数方法依赖稀疏结果奖励,无法认可部分正确解法中的中间正确步骤。过程奖励模型(PRMs)提供细粒度的步骤级监督,但其评分常存在噪声且难以评估。因此,近期PRM基准侧重于客观能力:检测推理路径中的首个错误步骤。然而,这一评估目标与PRMs在RL中的实际使用方式不一致——通常将其步骤得分作为原始奖励进行最大化。为弥合这一差距,我们提出可验证前缀策略优化(VPPO),仅在强化学习中利用PRMs定位首个错误。对于错误轨迹,VPPO将序列划分为已验证的正确前缀和错误后缀,仅奖励前者,并在检测到错误后施加针对性惩罚。该设计产生稳定、可解释的学习信号,显著改善信用分配。在多个推理基准测试中,VPPO在Pass@1和Pass@K指标上均持续优于稀疏奖励强化学习及先前基于PRM的基线方法。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs). However, most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions. Process reward models (PRMs) offer fine-grained step-level supervision, but their scores are often noisy and difficult to evaluate. As a result, recent PRM benchmarks focus on a more objective capability: detecting the first incorrect step in a reasoning path. However, this evaluation target is misaligned with how PRMs are typically used in RL, where their step-wise scores are treated as raw rewards to maximize. To bridge this gap, we propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL. Given an incorrect rollout, VPPO partitions the trajectory into a verified correct prefix and an erroneous suffix based on the first error, rewarding the former while applying targeted penalties only after the detected mistake. This design yields stable, interpretable learning signals and improves credit assignment. Across multiple reasoning benchmarks, VPPO consistently outperforms sparse-reward RL and prior PRM-guided baselines on both Pass@1 and Pass@K.

强化学习推理增强信用分配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。