BRIT通过统一图文关系图，实现跨模态多跳检索增强生成。
BRIT: Bidirectional Retrieval over Unified Image-Text Graph
- 构建图文统一图结构，双向遍历文本与图像路径
- 在多模态问答测试集上显著优于基线模型
- 适合处理需理解图文关联的复杂跨模态问题
检索增强生成（RAG）已成为提升大语言模型生成质量与相关性的有效方法。尽管近期进展主要聚焦于文本查询的RAG优化，但针对包含文本与图像的多模态文档的RAG仍未充分探索，尤其在无法微调的情况下。本文提出BRIT，一种新颖的多模态RAG框架，将文档中的各类图文关联统一为多模态图，并根据查询检索出特定子图。通过同时遍历图像到文本和文本到图像的路径，BRIT不仅能获取直接相关的图文内容，还能进一步挖掘回答复杂跨模态多跳问题所需的深层信息。为评估BRIT有效性，我们构建了专门针对多模态问答任务的MM-RAG测试集，该任务要求理解图文关系。全面实验表明，BRIT在多模态文档上处理跨模态问题方面具有显著优势。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has emerged as a promising technique to enhance the quality and relevance of responses generated by large language models. While recent advancements have mainly focused on improving RAG for text-based queries, RAG on multi-modal documents containing both texts and images has not been fully explored. Especially when fine-tuning does not work. This paper proposes BRIT, a novel multi-modal RAG framework that effectively unifies various text-image connections in the document into a multi-modal graph and retrieves the texts and images as a query-specific sub-graph. By traversing both image-to-text and text-to-image paths in the graph, BRIT retrieve not only directly query-relevant images and texts but also further relevant contents to answering complex cross-modal multi-hop questions. To evaluate the effectiveness of BRIT, we introduce MM-RAG test set specifically designed for multi-modal question answering tasks that require to understand the text-image relations. Our comprehensive experiments demonstrate the superiority of BRIT, highlighting its ability to handle cross-modal questions on the multi-modal documents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。