用结构化图规划选择关键多模态证据,提升问答准确性
HANIA: Planner-Guided Multimodal Graph Evidence Selection for Grounded Question Answering

- 通过规划器协调描述与关系证据,构建多模态图
- 在ScienceQA上准确率达78.3%,且无需微调或迭代检索
- 适合需要高可靠证据选择的多模态问答场景
多模态问答仍易受噪声、不完整和弱关联证据影响。长文本上下文会引入冗余并导致无依据生成,而平面检索可能忽略多步推理所需的关系。我们提出HANIA,一种基于规划器引导的多模态图证据选择框架。HANIA利用冻结的视觉语言模型处理图像与文本,提取简洁的、带明确弃权机制的视觉相关证据。随后构建输入相关的多模态图,并应用双组有限状态规划器协调描述性与关系性证据。覆盖率感知剪枝基于相关性、图置信度、概念覆盖度和模态多样性保留紧凑证据集。选定的段落、视觉陈述和图三元组被输入至冻结的指令微调解码器。我们在ScienceQA上评估了HANIA的准确率、证据过滤质量、证据预算敏感性和效率。结果表明,结构化证据规划与紧凑图引导检索可在无需目标数据集微调或迭代检索的情况下支持具有竞争力的多模态问答。代码已开源。
原文摘要 · Abstract (English)
Multimodal question answering remains sensitive to noisy, incomplete, and weakly grounded evidence. Long unstructured contexts can introduce redundancy and encourage unsupported generation, while flat retrieval may overlook relations needed for multi-step reasoning. We present HANIA, a planner-guided multimodal graph framework for evidence-grounded question answering. HANIA processes the supplied image and text using a frozen vision-language model to extract concise question-relevant visual evidence with explicit abstention. It then constructs an input-grounded multimodal graph and applies a two-group finite-state planner to coordinate descriptive and relational evidence. Coverage-aware pruning retains a compact evidence set based on relevance, graph confidence, concept coverage, and modality diversity. The selected passages, visual statements, and graph triples are provided to a frozen instruction-tuned decoder. We evaluate HANIA on ScienceQA using answer accuracy, evidence-filtering quality, evidence-budget sensitivity, and efficiency. The results show that structured evidence planning and compact graph-guided retrieval can support competitive multimodal question answering without target-dataset fine-tuning or iterative retrieval. The code is available at https://github.com/Zafar-southeast/HANIA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。