提升视觉空间推理的可信度,让模型思考过程更符合图像事实。
Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization
- 通过约束优化确保推理链与答案一致、描述贴合图像细节。
- 使推理不一致率从24.5%降至1.7%,图像关联得分提升13%。
- 适合关注模型可解释性与高精度视觉推理的研究者使用。
采用可验证奖励的强化学习训练的多模态推理模型(MRMs)在视觉推理基准上表现更优,但其准确率提升常伴随推理质量下降:生成的思维链(CoT)与最终答案不一致,且与图像证据脱节。我们在七个真实世界的空间推理基准上系统研究该现象,发现包括ViGoRL-Spatial、TreeVGR在内的主流模型均受影响。我们从逻辑一致性(推理是否蕴含答案)和视觉扎根性(每步是否准确描述图像中对象、属性与空间关系)两个维度评估推理质量。为此提出忠实的组相对策略优化(FGRPO),通过拉格朗日对偶上升法将一致性与扎根性作为约束嵌入优势计算,自适应调整约束权重。在Qwen2.5-VL-7B和3B模型上测试,FGRPO显著提升推理质量:不一致率由24.5%降至1.7%,视觉扎根性得分提升+13%,同时提升最终答案准确率,证明可信推理能带来更好结果。
原文摘要 · Abstract (English)
Multimodal reasoning models (MRMs) trained with reinforcement learning with verifiable rewards (RLVR) show improved accuracy on visual reasoning benchmarks. However, we observe that accuracy gains often come at the cost of reasoning quality: generated Chain-of-Thought (CoT) traces are frequently inconsistent with the final answer and poorly grounded in the visual evidence. We systematically study this phenomenon across seven challenging real-world spatial reasoning benchmarks and find that it affects contemporary MRMs such as ViGoRL-Spatial, TreeVGR as well as our own models trained with standard Group Relative Policy Optimization (GRPO). We characterize CoT reasoning quality along two complementary axes: "logical consistency" (does the CoT entail the final answer?) and "visual grounding" (does each reasoning step accurately describe objects, attributes, and spatial relationships in the image?). To address this, we propose Faithful GRPO (FGRPO), a variant of GRPO that enforces consistency and grounding as constraints via Lagrangian dual ascent. FGRPO incorporates batch-level consistency and grounding constraints into the advantage computation within a group, adaptively adjusting the relative importance of constraints during optimization. We evaluate FGRPO on Qwen2.5-VL-7B and 3B backbones across seven spatial datasets. Our results show that FGRPO substantially improves reasoning quality, reducing the inconsistency rate from 24.5% to 1.7% and improving visual grounding scores by +13%. It also improves final answer accuracy over simple GRPO, demonstrating that faithful reasoning enables better answers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。