通过三阶段流程提升视觉问答中的知识检索质量
Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
- 分三阶段动态处理、检索与过滤多模态知识
- 在E-VQA和InfoSeek上准确率分别提升36.0和42.8
- 适合需要高精度知识融合的视觉问答场景
基于知识的视觉问答（KB-VQA）要求视觉语言模型（VLMs）结合视觉理解与外部知识检索。尽管检索增强生成（RAG）通过知识库查询取得进展，但仍面临多模态查询质量低、检索结果相关性不足的问题。为此，我们提出一种三阶段方法Wiki-PRF，包括处理、检索和过滤。处理阶段动态调用视觉工具提取精确多模态信息；检索阶段融合视觉与文本特征实现多模态知识检索；过滤阶段进行相关性筛选与结果聚焦。我们引入一种以答案准确率和格式一致性为奖励信号的视觉语言模型，通过强化学习提升推理能力、工具调用准确性及无关内容过滤能力。在基准数据集E-VQA和InfoSeek上的实验表明，答案质量显著提升（分别达36.0和42.8），达到当前最优性能。代码已开源。
原文摘要 · Abstract (English)
Knowledge-based visual question answering (KB-VQA) requires visual language models (VLMs) to integrate visual understanding with external knowledge retrieval. Although retrieval-augmented generation (RAG) achieves significant advances in this task by combining knowledge-base querying, it still struggles with the quality of multimodal queries and the relevance of retrieved results. To overcome these challenges, we propose a novel three-stage method, termed Wiki-PRF, including Processing, Retrieval and Filtering stages. The processing stage dynamically invokes visual tools to extract precise multimodal information for retrieval. The retrieval stage integrates visual and text features to achieve multimodal knowledge retrieval. The filtering stage performs relevance filtering and concentration on retrieval results. To this end, we introduce a visual language model trained with answer accuracy and format consistency as reward signals via a reinforcement learning manner. This enhances the model's reasoning, tool invocation for accurate queries, and filtering of irrelevant content. Experiments on benchmark datasets (E-VQA and InfoSeek) show significant improvements~(36.0 and 42.8) in answer quality, achieving state-of-the-art performance. Code is available at https://github.com/cqu-student/Wiki-PRF
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。