arXiv:2607.21155cs.CVcs.AI2026-07中稿 · ECCV

拆解视觉问答中知识获取的每一步,找出模型瓶颈所在。

CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

论文配图:CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA
图 1 · 摘自论文原文
  • 按阶段标注数据,分离语言定位、物体识别与知识检索
  • 发现知识检索与推理是主要瓶颈,物体识别也存在问题
  • 适合研究多模态模型缺陷或改进评估方法的开发者

知识密集型视觉问答(KI-VQA)基准通过要求模型在图像之外调用外部知识来回答问题,评估视觉语言模型(VLMs)作为多模态知识助手的能力。KI-VQA涉及多个子任务——指代表达理解、视觉定位、物体识别、知识检索和推理——但现有基准仅报告最终准确率,掩盖了失败的具体环节。为此,我们提出CRAG-MM-Diagnostics诊断基准,包含分阶段的数据标注,可分离1)基于语言的视觉定位、2)物体识别、3)知识检索与推理。我们评估了全参数化和检索增强型VLMs,利用新收集的元数据(如目标感兴趣区域、实体名称、视觉复杂度评分)进行细粒度分析。结果表明,知识检索与推理是主要瓶颈,同时揭示模型在目标物体识别上存在困难,以及图像检索器难以融合文本线索的问题。这些发现暴露了当前KI-VQA系统的根本局限,推动阶段感知式评估。最后,基于此洞察,我们提出一种结合视觉定位模块的双模态RAG流水线,在图像检索前裁剪目标区域,使GPT-5和Qwen的准确率分别提升13.3和8.5个百分点。

原文摘要 · Abstract (English)

Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants by requiring external information beyond a provided image to answer questions. KI-VQA involves multiple sub-problems -referring expression understanding, visual grounding, object recognition, knowledge retrieval, and reasoning-yet existing benchmarks typically report only end-task accuracy, obscuring where failures arise. To analyze the full KI-VQA pipeline, we introduce CRAG-MM-Diagnostics, a diagnostic benchmark with stage-wise data annotations that isolate 1) language-based visual grounding, 2) object identification, and 3) knowledge retrieval and reasoning. We evaluate fully parametric and retrieval-augmented VLMs, providing fine-grained analyses using newly collected metadata, such as target ROIs, entity names, and visual complexity scores. Our results point to knowledge retrieval and reasoning as the primary bottleneck, but also highlight issues in the other parts of the KI-VQA pipeline, such as the fact that VLMs struggle with target object identification or that image retrievers struggle to integrate textual cues. These findings expose fundamental limitations in current KI-VQA systems and motivate stage-aware evaluation. We, lastly, leverage these findings to propose a grounded bimodal RAG pipeline that integrates a visual grounding module to crop targets before image retrieval, boosting GPT-5 and Qwen's respective accuracies by 13.3 and 8.5 percentage points.

视觉问答知识检索模型诊断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。