arXiv:2506.11991cs.CVcs.AI2025-06被引 61

让大模型先看图再推理,提升视觉细节理解能力。

VGR: Visual Grounded Reasoning

  • 先定位图像关键区域,再结合视觉信息推理
  • 用30%图像标记实现更优的多模态任务表现
  • 适合需要精细视觉分析的复杂推理场景

在多模态思维链(CoT)推理领域,现有方法主要依赖纯语言空间推理,存在语言偏见,且局限于数学与科学领域,难以处理需深入理解图像细节的复杂任务。为此,本文提出VGR,一种具备细粒度视觉感知能力的多模态大语言模型。与仅在语言空间作答的传统模型不同,VGR先检测可能有助于解题的图像区域,并在重播阶段将这些区域整合进推理过程。为此,构建了大规模SFT数据集VGR-SFT,包含混合视觉引导与语言推断的推理数据。在LLaVA-NeXT-7B基线上的实验表明,VGR在需全面理解图像细节的多模态基准上表现优异:仅使用30%的图像标记,就在MMStar上提升4.1分、AI2D上提升7.1分、ChartQA上提升12.9分。

原文摘要 · Abstract (English)

In the field of multimodal chain-of-thought (CoT) reasoning, existing approaches predominantly rely on reasoning on pure language space, which inherently suffers from language bias and is largely confined to math or science domains. This narrow focus limits their ability to handle complex visual reasoning tasks that demand comprehensive understanding of image details. To address these limitations, this paper introduces VGR, a novel reasoning multimodal large language model (MLLM) with enhanced fine-grained visual perception capabilities. Unlike traditional MLLMs that answer the question or reasoning solely on the language space, our VGR first detects relevant regions that may help to solve problems, and then provides precise answers based on replayed image regions. To achieve this, we conduct a large-scale SFT dataset called VGR -SFT that contains reasoning data with mixed vision grounding and language deduction. The inference pipeline of VGR allows the model to choose bounding boxes for visual reference and a replay stage is introduced to integrates the corresponding regions into the reasoning process, enhancing multimodel comprehension. Experiments on the LLaVA-NeXT-7B baseline show that VGR achieves superior performance on multi-modal benchmarks requiring comprehensive image detail understanding. Compared to the baseline, VGR uses only 30\% of the image token count while delivering scores of +4.1 on MMStar, +7.1 on AI2D, and a +12.9 improvement on ChartQA.

多模态推理视觉理解大模型图像标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。