让大模型既能检索图片又能生成解释，一次完成问答与推理。
UniCoRN: Unified Commented Retrieval Network with LMMs
- 用实体适配器把检索到的图像注入大模型，使其能结合视觉内容生成答案。
- 在复杂查询上比当前最好方法召回率高4.5%，评论生成质量提升14.9%以上。
- 适合需要图文理解、多步推理和自然语言解释的应用场景。
多模态检索方法在处理需推理视觉内容的复合查询时存在局限。大型多模态模型（LMMs）虽能回答复杂视觉问题，但缺乏检索相关实体的能力。为此，我们提出统一注释检索网络（UniCoRN），融合复合多模态检索与生成式语言方法，超越检索增强生成（RAG）。引入实体适配器模块，将检索到的多模态实体重新注入LMM，使其在生成答案和注释时可关注这些实体。通过冻结基础LMM，UniCoRN保持其原有能力，同时在统一框架下实现检索与文本生成。为评估新能力，我们提出注释检索任务（CoR）及对应数据集，目标是检索准确回答问题的图像并生成补充说明的文本。实验表明，UniCoRN在多个数据集上表现优异：复合多模态检索召回率较现有最优方法提升4.5%；在CoR任务中，评论生成的METEOR得分提高14.9%，BEM得分提高18.4%。
原文摘要 · Abstract (English)
Multimodal retrieval methods have limitations in handling complex, compositional queries that require reasoning about the visual content of both the query and the retrieved entities. On the other hand, Large Multimodal Models (LMMs) can answer with language to more complex visual questions, but without the inherent ability to retrieve relevant entities to support their answers. We aim to address these limitations with UniCoRN, a Unified Commented Retrieval Network that combines the strengths of composed multimodal retrieval methods and generative language approaches, going beyond Retrieval-Augmented Generation (RAG). We introduce an entity adapter module to inject the retrieved multimodal entities back into the LMM, so it can attend to them while generating answers and comments. By keeping the base LMM frozen, UniCoRN preserves its original capabilities while being able to perform both retrieval and text generation tasks under a single integrated framework. To assess these new abilities, we introduce the Commented Retrieval task (CoR) and a corresponding dataset, with the goal of retrieving an image that accurately answers a given question and generate an additional textual response that provides further clarification and details about the visual information. We demonstrate the effectiveness of UniCoRN on several datasets showing improvements of +4.5% recall over the state of the art for composed multimodal retrieval and of +14.9% METEOR / +18.4% BEM over RAG for commenting in CoR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。