arXiv:2506.16141cs.CVcs.AI2025-06ACL被引 51

提升多模态模型推理一致性,避免答对但逻辑混乱。

GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning

  • 用双层奖励机制,同时优化答案正确性和推理连贯性。
  • 在最难场景下准确率提升6.7%,推理一致性提高24.5%。
  • 适合关注多模态模型可解释性与鲁棒性的研究者。

近期强化学习方法如结果监督的GRPO已推动大语言模型的思维链推理,但其在多模态大语言模型(MLLMs)中的应用尚未探索。为弥补MLLM后训练方法评估的不足,我们提出SEED-Bench-R1基准,包含需平衡感知与推理的真实世界视频,涵盖三个递增挑战:分布内、跨环境、跨环境-任务。基于该基准,发现标准GRPO虽提升答案准确率,却使推理步骤与答案的一致性降至57.9%,源于仅依赖最终答案的奖励信号导致捷径学习,且严格KL惩罚抑制探索。为此,我们提出GRPO-CARE,一种一致性感知的强化学习框架,在无需显式监督下同时优化答案正确性与推理连贯性。该框架采用双层奖励:(1) 基础奖励保障答案正确;(2) 自适应一致性奖励,通过慢变参考模型比较模型推理到答案的似然度与群体同伴。该机制放大既正确又一致的推理路径奖励。以自适应奖励替代KL惩罚,GRPO-CARE在SEED-Bench-R1上实现6.7%性能提升(最难点),一致性提升24.5%,并展现强迁移能力,显著改善多个视频理解基准表现。本工作贡献了系统化设计的基准与通用后训练框架,推动更可解释、鲁棒的MLLM发展。

原文摘要 · Abstract (English)

Recent reinforcement learning approaches, such as outcome-supervised GRPO, have advanced Chain-of-Thought reasoning in large language models (LLMs), yet their adaptation to multimodal LLMs (MLLMs) is unexplored. To address the lack of rigorous evaluation for MLLM post-training methods, we introduce SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced perception and reasoning. It offers a large training set and evaluates generalization across three escalating challenges: in-distribution, cross-environment, and cross-environment-task scenarios. Using SEED-Bench-R1, we find that standard GRPO, while improving answer accuracy, often reduces logical coherence between reasoning steps and answers, with only a 57.9% consistency rate. This stems from reward signals focusing solely on final answers, encouraging shortcuts, and strict KL penalties limiting exploration.To address this, we propose GRPO-CARE, a consistency-aware RL framework optimizing both answer correctness and reasoning coherence without explicit supervision. GRPO-CARE introduces a two-tiered reward: (1) a base reward for answer correctness, and (2) an adaptive consistency bonus, computed by comparing the model's reasoning-to-answer likelihood (via a slowly-evolving reference model) against group peers.This dual mechanism amplifies rewards for reasoning paths that are both correct and logically consistent. Replacing KL penalties with this adaptive bonus, GRPO-CARE outperforms standard GRPO on SEED-Bench-R1, achieving a 6.7% performance gain on the hardest evaluation level and a 24.5% improvement in consistency. It also shows strong transferability, improving model performance across diverse video understanding benchmarks. Our work contributes a systematically designed benchmark and a generalizable post-training framework, advancing the development of more interpretable and robust MLLMs.

多模态强化学习推理一致性后训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。