让视觉问答模型学会常识推理，提升真实场景下的理解能力。
MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering
- 三阶段融合外部常识知识与视觉语言模型
- 图神经网络增强关系推理，准确率显著提升
- 无需复杂微调，适合实际部署的常识推理任务
视觉问答（VQA）需要跨视觉与文本模态进行推理，但大型视觉语言模型（LVLMs）常缺乏整合的常识知识，限制了其在现实场景中的鲁棒性。为此，我们提出MAGIC-VQA框架，通过系统性整合常识知识来增强VQA性能。该框架采用三阶段流程：(1) 从外部来源显式引入常识知识，(2) 按类型进行后处理以实现上下文优化，(3) 利用图神经网络（GNN）进行隐式知识增强，实现结构化推理。尽管GNN带来更深层次的结构推理能力，但仍能超越仅依赖LVLM的模型。MAGIC-VQA通过统一常识知识与基于LVLM的推理，避免了大量预训练或复杂提示调优的需求。在基准数据集上达到领先性能，显著提升了VQA中的常识推理能力。
原文摘要 · Abstract (English)
Visual Question Answering (VQA) requires reasoning across visual and textual modalities, yet Large Vision-Language Models (LVLMs) often lack integrated commonsense knowledge, limiting their robustness in real-world scenarios. To address this, we introduce MAGIC-VQA, a novel framework that enhances VQA by systematically integrating commonsense knowledge with LVLMs. MAGIC-VQA employs a three-stage process: (1) Explicit Knowledge Integration from external sources, (2) By-Type Post-Processing for contextual refinement, and (3) Implicit Knowledge Augmentation using a Graph Neural Network (GNN) for structured reasoning. While GNNs bring greater depth to structured inference, they enable superior relational inference beyond LVLMs. MAGIC-VQA bridges a key gap by unifying commonsensse knowledge with LVLM-driven reasoning, eliminating the need for extensive pre-training or complex prompt tuning. Our framework achieves state-of-the-art performance on benchmark datasets, significantly improving commonsense reasoning in VQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。