用过程一致性筛选数据，让模型既答对又会推理。
Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training
- 通过过程与结果一致性筛选训练数据，避免奖励欺骗
- 在多个基准上同时提升答案准确率和推理质量
- 不依赖强过程奖励模型，适合实际部署
基于可验证奖励的强化学习（RLVR）虽能提升推理任务的答案准确率，但无法可靠改善推理质量。由于结果奖励仅评估最终答案，会奖励那些过程有误却偶然得出正确结果的情况，导致梯度偏差，使当前方法难以学习忠实的推理过程。过程奖励模型（PRMs）提供步骤级监督，但在强化学习过程中直接优化或简单融合过程与结果奖励，在分布偏移下极不稳定。本文提出过程一致性过滤器（PROF），一种基于PRM与ORM一致性的数据筛选方法，而非直接优化奖励。PROF保留过程支持强的正确答案与过程支持弱的错误答案，同时保持训练样本比例均衡。实验表明，PROF在多个强基线上持续提升最终答案准确率与中间推理质量，且对强过程奖励模型依赖更低。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) improves final-answer accuracy on reasoning tasks, but it does not reliably improve reasoning quality. Because outcome rewards only assess final answers, they also reward spurious successes: flawed reasoning can still receive maximal reward when it accidentally reaches the correct outcome. This outcome reward hacking creates biased gradients, making current RLVR insufficient for learning faithful reasoning. Process Reward Models (PRMs) provide step-wise supervision, but directly optimizing PRMs or naively combining them with outcome rewards is unstable under distribution shift during RL training process. We introduce PRocess cOnsistency Filter (PROF), a data curation method that uses PRM--ORM consistency for sample selection rather than direct reward optimization. PROF keeps correct responses with strong process support and incorrect responses with weak process support while maintaining a balanced training ratio. Experiments show that PROF consistently improves both final-answer accuracy and intermediate reasoning quality over strong baselines, with less dependence on strong PRMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。