通过筛选视觉与外部知识，提升多模态问答准确率。
MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering
- 先用掩码选择机制剔除无关图像区域和弱相关知识
- 在多个基准上相比基线提升答案准确率，效果稳定
- 适合需要精准推理的多模态知识问答场景
基于知识的视觉问答（KB-VQA）要求模型结合视觉信息与外部知识作答。然而，检索到的知识常含噪声、部分无关或与视觉内容不匹配，而模型内部知识难以控制与解释。简单拼接两类知识会限制推理效果并降低准确率。为此，我们提出MaS-VQA，一种以选择驱动的框架，将显式知识过滤与隐式知识推理紧密耦合。该框架首先检索候选文本段落，再通过掩码-选择机制联合剔除无关图像区域和弱相关知识片段，生成紧凑且高信噪比的多模态知识。这些过滤后的知识进一步约束内部知识在有限语义空间中的激活，实现显式与隐式知识的互补协同建模，从而提升答案预测鲁棒性。在Encyclopedic-VQA与InfoSeek数据集上的实验表明，该框架在多种大语言模型骨干网络下均取得持续性能提升，消融实验验证了选择机制有效降低噪声并增强知识利用率。
原文摘要 · Abstract (English)
Knowledge-based Visual Question Answering (KB-VQA) requires models to answer questions by integrating visual information with external knowledge. However, retrieved knowledge is often noisy, partially irrelevant, or misaligned with the visual content, while internal model knowledge is difficult to control and interpret. Naive aggregation of these sources limits reasoning effectiveness and reduces answer accuracy. To address this, we propose MaS-VQA, a selection-driven framework that tightly couples explicit knowledge filtering with implicit knowledge reasoning. MaS-VQA first retrieves candidate passages and applies a Mask-and-Select mechanism to jointly prune irrelevant image regions and weakly relevant knowledge fragments, producing compact, high-signal multimodal knowledge . This filtered knowledge then guides the activation of internal knowledge in a constrained semantic space, enabling complementary co-modeling of explicit and implicit knowledge for robust answer prediction. Experiments on Encyclopedic-VQA and InfoSeek demonstrate consistent performance gains across multiple MLLM backbones, and ablations verify that the selection mechanism effectively reduces noise and enhances knowledge utilization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。