arXiv:2605.10325cs.AI2026-05被引 6

让大模型推理过程可验证,提升长程思考的准确性。

Verifiable Process Rewards for Agentic Reasoning

论文配图:Verifiable Process Rewards for Agentic Reasoning
图 1 · 摘自论文原文
  • 用符号或算法验算器生成每一步的奖励信号,替代稀疏结果反馈。
  • 在三种推理场景中,性能超越仅依赖最终结果的基线方法。
  • 适合需要可靠中间验证的复杂推理任务,如逻辑推演与概率分析。

基于可验证奖励的强化学习(RLVR)提升了大语言模型的推理能力,但现有方法多依赖稀疏的结果级反馈。这种稀疏性导致长程代理推理中的信用分配难题:一条轨迹可能因多个正确中间决策而失败,或因存在错误却仍成功。本文研究一类可密集验证的代理推理问题,其中中间动作可通过符号或算法验算器客观检验。我们提出可验证过程奖励(VPR)框架,将此类验算器转化为强化学习中的密集回合级监督信号,并在三种典型场景中实现:基于搜索的动态演绎验证、基于约束的逻辑推理验证,以及基于后验的概率推理验证。理论分析表明,密集验证器驱动的奖励能通过更局部的学习信号改善长程信用分配,其收益取决于验证器可靠性。实验显示,VPR在受控环境中优于结果级奖励和基于回溯的流程奖励基线,并成功迁移至通用与代理推理基准,表明可验证过程监督可培养超越训练环境的泛化推理能力。结果表明,当存在可靠中间验证时,VPR是增强大模型代理的有效途径,但也凸显其对验证器质量的依赖,以及向非结构化开放环境扩展的挑战。

原文摘要 · Abstract (English)

Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of large language models (LLMs), but most existing approaches rely on sparse outcome-level feedback. This sparsity creates a credit assignment challenge in long-horizon agentic reasoning: a trajectory may fail despite containing many correct intermediate decisions, or succeed despite containing flawed ones. In this work, we study a class of densely-verifiable agentic reasoning problems, where intermediate actions can be objectively checked by symbolic or algorithmic oracles. We propose Verifiable Process Rewards (VPR), a framework that converts such oracles into dense turn-level supervision for reinforcement learning, and instantiate it in three representative settings: search-based verification for dynamic deduction, constraint-based verification for logical reasoning, and posterior-based verification for probabilistic inference. We further provide a theoretical analysis showing that dense verifier-grounded rewards can improve long-horizon credit assignment by providing more localized learning signals, with the benefit depending on the reliability of the verifier. Empirically, VPR outperforms outcome-level reward and rollout-based process reward baselines across controlled environments, and more importantly, transfers to both general and agentic reasoning benchmarks, suggesting that verifiable process supervision can foster general reasoning skills applicable beyond the training environments. Our results indicate that VPR is a promising approach for enhancing LLM agents whenever reliable intermediate verification is available, while also highlighting its dependence on oracle quality and the open challenge of extending VPR to less structured, open-ended environments.

强化学习推理能力可验证性大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。