系统梳理视觉问答的四大数据集与六大模型方法,助你快速掌握领域全貌。
A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
- 按真实/合成/诊断/知识库四类划分数据集,清晰分类
- 总结融合、注意力、外部知识等六种核心模型范式
- 适合入门者快速了解VQA研究脉络与关键挑战
视觉问答(VQA)要求模型根据图像和自然语言问题生成正确答案,需同时理解视觉内容与语言语义,并具备推理能力。自该领域兴起以来,大量VQA数据集与模型被提出。本文系统分析当前VQA数据集与模型现状,将其明确分为四类:(1) 包含丰富真实图像的公开数据集;(2) 仅由人工生成的合成图像数据集;(3) 专门用于测试特定能力(如场景文本理解)的诊断数据集;(4) 用于评估模型利用外部知识能力的知识库(KB)数据集。同时,我们探讨六种主要模型范式:融合(多模态信息整合)、注意力(跨模态信息筛选)、外部知识库(使用外部信息)、组合或推理(处理需复杂推理的问题)、解释生成(生成视觉与文本描述以验证推理过程),以及图模型(通过图结构编码与操作关系)。此外还讨论了场景文本理解、计数与偏差减少等相关议题。
原文摘要 · Abstract (English)
Visual question answering (VQA) refers to the problem where, given an image and a natural language question about the image, a correct natural language answer has to be generated. A VQA model has to demonstrate both the visual understanding of the image and the semantic understanding of the question, demonstrating reasoning capability. Since the inception of this field, a plethora of VQA datasets and models have been published. In this article, we meticulously analyze the current state of VQA datasets and models, while cleanly dividing them into distinct categories and then summarizing the methodologies and characteristics of each category. We divide VQA datasets into four categories: (1) available datasets that contain a rich collection of authentic images, (2) synthetic datasets that contain only synthetic images produced through artificial means, (3) diagnostic datasets that are specially designed to test model performance in a particular area, e.g., understanding the scene text, and (4) KB (Knowledge-Based) datasets that are designed to measure a model's ability to utilize outside knowledge. Concurrently, we explore six main paradigms of VQA models: fusion, where we discuss different methods of fusing information between visual and textual modalities; attention, the technique of using information from one modality to filter information from another; external knowledge base, where we discuss different models utilizing outside information; composition or reasoning, where we analyze techniques to answer advanced questions that require complex reasoning steps; explanation, which is the process of generating visual and textual descriptions to verify sound reasoning; and graph models, which encode and manipulate relationships through nodes in a graph. We also discuss some miscellaneous topics, such as scene text understanding, counting, and bias reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。