通过排序奖励提升多模态推理可靠性，解决答案正确但推理无效的问题。
Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness
- 采用分组排序奖励机制，一次性比较同一问题的多个推理路径。
- 在多模态任务中将可靠推理准确率从47.4%提升至54.7%。
- 适合关注推理过程可信度的研究者或需要高可靠性AI系统的设计者。
强化学习结合可验证奖励（RLVR）通过奖励可验证的最终答案来提升多模态推理能力。然而，仅答案正确并不意味着推理过程可靠，部分轨迹可能依赖不完整推导、弱证据或自相矛盾的陈述，这种‘推理-答案不一致’现象阻碍了模型真正理解。为解决此问题，本文对比了奖励模型（RMs）与生成式奖励（GRs）两种轨迹监督方法：前者训练初期高效但后期效果衰减；后者性能更优但存在奖励不稳定与计算开销大问题。为此提出分组排序奖励（Groupwise Ranking Reward），对同一提示下通过验证器的多个轨迹进行一次性排序并重新分配奖励，相比GRs显著降低评估成本且更好区分强弱正确路径。实验表明，传统RLVR会加剧推理-答案不一致，而轨迹监督可有效缓解该问题。分组排序奖励表现最优，在可靠性条件下的准确率由47.4%提升至54.7%。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) improves multimodal reasoning by rewarding verifiable final answers. Yet answer-correct trajectories may still rely on incomplete derivations, weak evidence, or statements that contradict their conclusions. This gap between answer correctness and reasoning validity, which we call reasoning-answer inconsistency, motivates trajectory supervision in multimodal RL. We compare two main approaches: reward models (RMs), and Generative Rewards (GRs). RMs are efficient and help early in training, but their gains weaken as the policy distribution shifts; GRs improve performance, but may give unstable rewards and computationally expensive. We therefore propose Groupwise Ranking Reward, which ranks verifier-passed trajectories for the same prompt in one pass and redistributes reward accordingly. Groupwise comparison better separates stronger and weaker correct trajectories with lower judge overhead than GRs. Experiments show that RLVR aggravates reasoning-answer inconsistency, while trajectory supervision alleviates it. Groupwise Ranking Reward performs best overall, improving reliability-conditioned accuracy from 47.4% to 54.7% over RLVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。