arXiv:2604.21396cs.CVcs.AI2026-04中稿 · LREC 2026被引 1

构建可信赖视觉推理数据集,让模型每步推理都有图像证据支持。

VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought

论文配图:VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought
图 1 · 摘自论文原文
  • 自动三阶段流程将每步推理与图像区域精准对齐。
  • 在多个视觉语言模型上验证,推理准确率与逻辑一致性显著提升。
  • 适合关注模型可信度、可解释性的研究者使用。

大型视觉语言模型(LVLMs)的发展需要基于图像区域的精确推理,确保模型逻辑真实反映视觉证据。然而,现有数据集因人工标注成本高且推理步骤与图像区域缺乏显式对齐,限制了模型可信度的评估。为此,我们提出视觉接地链式思维(VG-CoT)数据集,通过全自动三阶段流程将每个推理步骤与图像中的真实视觉证据明确关联:首先利用先进检测与OCR模型提取物体和文本级视觉证据;其次用GPT-4o生成分步接地推理;最后通过基于理由的开集检测进行接地精炼。此外,我们引入一个新基准,从三个互补维度综合评估LVLMs的推理能力:推理质量、答案准确率和推理-答案对齐性。在代表性模型(如LLaVA-1.5和Qwen2-VL)上的实验表明,多数评估指标均有显著提升,证实VG-CoT能有效促进可信赖、基于证据的推理,同时实现高效低成本的数据构建。数据集与代码将在论文录用后公开,以推动后续研究。

原文摘要 · Abstract (English)

The advancement of Large Vision-Language Models (LVLMs) requires precise local region-based reasoning that faithfully grounds the model's logic in actual visual evidence. However, existing datasets face limitations in scalability due to extensive manual annotation and lack of explicit alignment between multi-step reasoning and corresponding image regions, which constrains the evaluation of model trustworthiness. To address these challenges, we propose the Visual Grounding Chain-of-Thought (VG-CoT) dataset, which explicitly links each reasoning step to real visual evidence within the image through a fully automated three-stage pipeline. The pipeline first extracts object- and text-level visual evidence using state-of-the-art detection and OCR models, then generates step-by-step grounded reasoning with GPT-4o, and finally refines the grounding through a rationale-driven open-set detection process. In addition, we introduce a new benchmark that comprehensively evaluates LVLMs reasoning across three complementary dimensions: Rationale Quality, Answer Accuracy, and Reasoning-Answer Alignment. Experiments with representative LVLMs, including LLaVA-1.5 and Qwen2-VL, demonstrate consistent improvements on most evaluation metrics, confirming that VG-CoT effectively enhances trustworthy, evidence-based reasoning while maintaining scalable and cost-efficient dataset construction. The dataset and code will be released publicly upon acceptance to facilitate further research.

视觉推理可解释性链式思维数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。