arXiv:2604.20659cs.LGcs.AI2026-04被引 2

通过可验证过程监督,提升大模型推理的精准与效率。

GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning

论文配图:GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
图 1 · 摘自论文原文
  • 用信念追踪方法监控每步推理正确性,实现无监督的过程监督。
  • 数学任务准确率最高提升2.6点,推理长度减少13.7%。
  • 无需额外模型或采样,适合追求高效推理的实践者。

基于可验证奖励的强化学习(RLVR)通过直接结果验证提升了大语言模型的推理能力。在此基础上,组相对策略优化(GRPO)虽无需价值函数模型,但对中间步骤的信用分配缺乏区分,限制了有效推理策略的识别并导致过度思考。本文提出一种无需模型的可验证过程监督机制,通过探测模型在推理轨迹中对正确答案的信念变化,将生成过程划分为离散步骤,跟踪每个步骤边界处正确答案的条件概率,从而高效计算可解释的段落级进展度量,用于优化GRPO的轨迹级反馈。该方法实现更精准、样本高效的策略更新,且无需依赖昂贵的蒙特卡洛回溯或辅助模型。在数学和通用领域基准测试中,性能均优于原GRPO:数学任务最高提升2.6点准确率,推理长度减少13.7%;通用任务最高提升2.4点,推理长度减少4%,展现出强泛化能力。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Language Models (LLMs) by leveraging direct outcome verification instead of learned reward models. Building on this paradigm, Group Relative Policy Optimization (GRPO) eliminates the need for critic models but suffers from indiscriminate credit assignment for intermediate steps, which limits its ability to identify effective reasoning strategies and incurs overthinking. In this work, we introduce a model-free and verifiable process supervision via probing the model's belief in the correct answer throughout its reasoning trajectory. By segmenting the generation into discrete steps and tracking the conditional probability of the correct answer appended at each segment boundary, we efficiently compute interpretable segment-wise progress measurements to refine GRPO's trajectory-level feedback. This approach enables more targeted and sample-efficient policy updates, while avoiding the need for intermediate supervision derived from costly Monte Carlo rollouts or auxiliary models. Experiments on mathematical and general-domain benchmarks show consistent gains over GRPO across diverse models: up to 2.6-point accuracy improvements and 13.7% reasoning-length reductions on math tasks, and up to 2.4 points and 4% on general-domain tasks, demonstrating strong generalization.

大模型推理强化学习过程监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。