arXiv:2506.07218cs.LGcs.AI2025-06被引 25

用视觉感知奖励提升多模态模型看图能力,让推理更准。

Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward

  • 设计视觉感知奖励,通过一致性判断激励模型准确理解图像。
  • 仅用1442条数据即在多个基准上达到顶尖表现。
  • 适合关注多模态推理与视觉理解结合的研究者。

提升多模态大语言模型(MLLMs)的多模态推理能力是当前研究热点。尽管已有工作采用可验证奖励强化学习(RLVR)来增强推理能力,但普遍忽视了对多模态感知能力的提升,而感知能力是复杂推理的基础。通过McNemar检验发现,现有RLVR方法无法有效提升MLLMs的视觉感知能力,从而限制了推理性能的进一步优化。为此,我们提出Perception-R1,引入一种新型视觉感知奖励,显式鼓励模型准确感知视觉内容,从而同时提升感知与推理能力。具体地,我们从多模态问题的思维链轨迹中收集文本化视觉注释,作为奖励分配的视觉参考。在RLVR训练中,利用判别性LLM评估模型输出与视觉注释的一致性,并据此分配感知奖励。大量实验表明,Perception-R1在多个多模态推理基准上表现优异,仅使用1,442条训练数据即达到当前最优水平。代码与数据集将开源于https://github.com/tongxiao2002/Perception-R1。

原文摘要 · Abstract (English)

Enhancing the multimodal reasoning capabilities of Multimodal Large Language Models (MLLMs) is a challenging task that has attracted increasing attention in the community. Recently, several studies have applied Reinforcement Learning with Verifiable Rewards (RLVR) to the multimodal domain in order to enhance the reasoning abilities of MLLMs. However, these works largely overlook the enhancement of multimodal perception capabilities in MLLMs, which serve as a core prerequisite and foundational component of complex multimodal reasoning. Through McNemar's test, we find that existing RLVR method fails to effectively enhance the multimodal perception capabilities of MLLMs, thereby limiting their further improvement in multimodal reasoning. To address this limitation, we propose Perception-R1, which introduces a novel visual perception reward that explicitly encourages MLLMs to perceive the visual content accurately, thereby can effectively incentivizing both their multimodal perception and reasoning capabilities. Specifically, we first collect textual visual annotations from the CoT trajectories of multimodal problems, which will serve as visual references for reward assignment. During RLVR training, we employ a judging LLM to assess the consistency between the visual annotations and the responses generated by MLLM, and assign the visual perception reward based on these consistency judgments. Extensive experiments on several multimodal reasoning benchmarks demonstrate the effectiveness of our Perception-R1, which achieves state-of-the-art performance on most benchmarks using only 1,442 training data. Our code and dataset will be available at https://github.com/tongxiao2002/Perception-R1.

多模态强化学习感知增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。