提出新方法让大模型在无答案标签下稳定提升推理能力
Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR
- 用掩码梯度方式分离奖励估计与策略优化
- 在多个基准上超越现有无标签方法，接近有监督效果
- 适合测试时训练场景，可稳定提升模型推理性能
基于可验证奖励的强化学习（RLVR）能提升大模型推理能力，但通常依赖真实答案（GT），限制了可扩展性。基于投票的无标签RLVR用模型采样结果的共识替代真实标签，但当同一答案信号同时用于奖励估计和策略优化时，会引发模型直接强化答案词元而非改进推理过程。本文提出OM-GRPO，一种解耦奖励估计与策略优化的无标签RLVR框架。该方法在答案片段上掩码梯度，保留答案级奖励通过软共识信号，将优化压力从答案词元转移开。进一步引入对比增强奖励，通过低成本的成对比较优化已有轨迹上的奖励估计，无需额外采样。在多种推理基准和三类LLM主干网络上，OM-GRPO持续优于现有无标签方法，并达到与有监督GT奖励训练相当的效果，优化过程更稳定。该稳定性在测试时训练（Test-Time Training）设置中尤为显著，相比多数投票方法提升4.24分。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability. Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples. However, collapse arises when the same answer-level signal is used both to estimate rewards and to drive token-level policy optimization, encouraging the model to directly reinforce answer tokens rather than improve reasoning. We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization. OM-GRPO masks gradients on the answer span while retaining answer-level rewards through a soft consensus signal, shifting optimization pressure away from answer tokens. We further introduce Contrast-Augmented Reward, which refines reward estimation via low-cost pairwise comparisons over existing trajectories without additional rollouts. Across diverse reasoning benchmarks and three LLM backbones, OM-GRPO consistently outperforms existing label-free RLVR methods and matches supervised GT-reward training with stable optimization. This stability is particularly beneficial in the Test-Time Training setting, where OM-GRPO surpasses majority voting by 4.24 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。