让AI回答问题时能指出依据区域,提升可解释性。
ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering
- 用问题感知的原型定位图像关键区域,连接答案与视觉证据。
- 在Visual7W数据集上实现高准确率,且解释与真实依据高度一致。
- 适合医疗、自动驾驶等需要可信推理的场景使用。
视觉问答(VQA)在从通用视觉推理到医疗影像、自动驾驶等安全关键领域应用日益广泛,要求模型不仅给出准确答案,还需提供人类可理解、可验证的解释。基于原型的建模在纯视觉推理任务中展现了良好的可解释性,通过将预测锚定在语义有意义的图像区域,但在VQA中仍研究不足。本文提出ProtoVQA,一个统一的原型框架:(i) 学习与问题相关的原型作为推理锚点,将答案与判别性图像区域关联;(ii) 采用空间约束匹配机制,确保所选证据在空间和语义上连贯;(iii) 通过共享原型主干网络同时支持问答与定位任务。为评估解释质量,提出视觉-语言对齐分数(VLAS),衡量模型关注区域与真实证据的一致性。在Visual7W数据集上的实验表明,ProtoVQA在保持竞争力准确率的同时,生成了忠实且细粒度的解释,推动透明、可信VQA系统的发展。
原文摘要 · Abstract (English)
Visual Question Answering (VQA) is increasingly used in diverse applications ranging from general visual reasoning to safety-critical domains such as medical imaging and autonomous systems, where models must provide not only accurate answers but also explanations that humans can easily understand and verify. Prototype-based modeling has shown promise for interpretability by grounding predictions in semantically meaningful regions for purely visual reasoning tasks, yet remains underexplored in the context of VQA. We present ProtoVQA, a unified prototypical framework that (i) learns question-aware prototypes that serve as reasoning anchors, connecting answers to discriminative image regions, (ii) applies spatially constrained matching to ensure that the selected evidence is coherent and semantically relevant, and (iii) supports both answering and grounding tasks through a shared prototype backbone. To assess explanation quality, we propose the Visual-Linguistic Alignment Score (VLAS), which measures how well the model's attended regions align with ground-truth evidence. Experiments on Visual7W show that ProtoVQA yields faithful, fine-grained explanations while maintaining competitive accuracy, advancing the development of transparent and trustworthy VQA systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。