让AI回答问题时精准定位证据并忠实推理,提升知识型视觉问答准确率。
SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering

- 通过多模态混合搜索和图结构细化检索,精准定位证据
- 引入基于证据的强化学习奖励,使答案正确必须依赖正确证据
- 在两个基准上分别提升8.9%和3.5%,适合需要可信推理的应用
知识型视觉问答(KB-VQA)旨在回答需要依赖视觉内容之外外部知识进行推理的问题。现有方法通常融合多模态特征检索外部信息,再利用多模态大模型基于检索到的证据生成答案。然而,这些方法往往难以捕捉复杂上下文中的结构关联,导致噪声过滤能力弱;同时,推理过程常脱离证据,缺乏忠实性。为此,本文提出SAFE-G框架,实现结构感知的精准证据定位与可信推理。首先采用粗粒度多模态混合搜索召回候选文档,再通过结构感知的细粒度图检索捕获结构依赖关系,有效过滤噪声并精确定位证据。此外,设计一种基于证据的强化学习策略,仅当所选证据正确时才给予答案奖励,强制模型将回答锚定在正确证据上,从而增强其通过多模态特征定位证据和忠实推理的能力。在Encyclopedic-VQA与InfoSeek两个基准上的实验表明,SAFE-G相比先前方法分别提升8.9%和3.5%,显著提高整体推理准确率。代码已开源:https://github.com/MINE-USTC/SAFE-G。
原文摘要 · Abstract (English)
Knowledge-based Visual Question Answering (KB-VQA) aims to answer queries that necessitate reasoning over external knowledge sources beyond the visual content. Typically, current methods fuse multimodal features to retrieve external information, subsequently leveraging Multimodal Large Language Models (MLLMs) to derive answers from the retrieved evidence. However, these methods often struggle to capture structural associations within complex contexts to effectively filter noise. Furthermore, they frequently fail to ensure that the reasoning process remains strictly faithful to the retrieved evidence. To address these challenges, we propose SAFE-G, a Structure-Aware Faithful Evidence-guided Generation framework, which enables precise evidence localization and trustworthy reasoning. Specifically, we first employ a coarse-grained hybrid search fusing visual and textual modalities to recall candidate documents, and subsequently implement a structure-aware fine-grained graph retrieval that captures structural dependencies to filter noise and pinpoint precise evidence. Moreover, we introduce a reinforcement learning (RL) strategy with an evidence-grounded reward that assigns credit to correct answers only when the selected evidence is correct. This strict alignment constraint compels the model to anchor its response in the retrieved context, effectively enhancing its capability to locate evidence via multimodal features and perform faithful reasoning. Extensive experiments on the Encyclopedic-VQA and InfoSeek benchmarks demonstrate that SAFE-G outperforms prior methods by a margin of 8.9% and 3.5%, substantially enhancing the overall reasoning accuracy. Our source code is publicly available at: https://github.com/MINE-USTC/SAFE-G.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。