将检索器内嵌到大模型中，实现视觉问答的端到端知识获取与生成。
Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
- 把检索功能直接集成到生成模型里，用自回归方式自动找相关文档。
- 在OKVQA和A-OKVQA上提升2.9%至9.6%，显著优于现有方法。
- 适合需要精准知识支持的视觉问答场景，尤其关注推理准确性。
检索增强生成（RAG）已成为解决知识密集型视觉问答（VQA）任务的重要方法。现有方法通常采用独立的检索模块和生成模块分别获取外部知识并生成答案。本文提出ReAuSE，一种替代传统RAG的新型模型，将知识检索器无缝嵌入生成式多模态大语言模型中，作为内置搜索引擎。该模型兼具生成式检索与精准答案生成能力：既能通过生成文档标识符从知识库中检索文档，又能基于检索结果回答视觉问题。此外，我们设计了基于相关性反馈的强化检索校准模块，进一步提升检索性能并契合准确答案生成的需求。在两个代表性数据集OKVQA和A-OKVQA上的大量实验表明，相较于强基线模型，各项评估指标均取得2.9%至9.6%的显著提升。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) has emerged to address the knowledge-intensive visual question answering (VQA) task. Current methods mainly employ separate retrieval and generation modules to acquire external knowledge and generate answers, respectively. We propose ReAuSE, an alternative to the previous RAG model for the knowledge-based VQA task, which seamlessly integrates knowledge retriever into the generative multi-modal large language model, serving as a built-in search engine. Specifically, our model functions both as a generative retriever and an accurate answer generator. It not only helps retrieve documents from the knowledge base by producing identifiers for each document, but it also answers visual questions based on the retrieved documents. Furthermore, we propose a reinforced retrieval calibration module from relevance feedback to improve retrieval performance and align with the preferences for accurate answer generation. Extensive experiments on two representative OKVQA and A-OKVQA datasets demonstrate significant improvements ranging from 2.9\% to 9.6\% across all evaluation metrics when compared to strong baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。