arXiv:2605.30451cs.LG2026-05被引 2

VeriGate让推理模型在缺乏反馈时仍能学习,提升准确率20%以上。

VeriGate: Verifier-Gated Step-Level Supervision for GRPO

论文配图:VeriGate: Verifier-Gated Step-Level Supervision for GRPO
图 1 · 摘自论文原文
  • 用验证器判断是否可用过程奖励,避免学习停滞
  • 将步骤奖励转为未来累积奖励,实现细粒度信用分配
  • 适合训练大模型推理能力,减少错误奖励干扰

Group Relative Policy Optimization(GRPO)是一种利用验证器输出奖励训练推理模型的有效方法,但其监督信号稀疏:当同一提示的所有采样轨迹获得相同验证器奖励时,组内相对优势归零,导致学习停滞。仅使用结果奖励也缺乏步骤级信用分配,限制探索并难以学习稳健推理。本文提出VeriGate(Verifier-Gated Step-Level GRPO),通过三项设计改进:首先,当验证器奖励能区分轨迹时由其主导,否则启用过程监督;其次,不将过程奖励模型(PRM)的步骤得分合并为单一轨迹奖励,而是转换为未来累积奖励,实现延续性信用分配;第三,将这些奖励转化为组归一化的词元级优势,恢复有效梯度与细粒度信用分配,且对奖励劫持更鲁棒。实验中,在MATH数据集上使用1.5B和7B的Qwen2.5-Instruct模型训练,并在六个推理基准测试评估,VeriGate使1.5B和7B模型平均准确率分别提升约20%和12%,显著减少零梯度失败,降低奖励劫持行为,优于仅使用结果的GRPO和将PRM作为结果的基线方法。

原文摘要 · Abstract (English)

Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.

推理模型强化学习验证器信用分配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。