构建图文表格统一查询图,提升长文档多模态问答的准确性和连贯性。
MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generation
- 以查询为中心构建跨模态跨页图结构,统一组织异构内容
- 在MMLongBench-Doc和LongDocURL上显著提升检索质量与答案准确率
- 适合需要精准理解长篇图文混合文档的研究与应用
理解包含段落、图表等多模态内容的长文档面临两大挑战:一是跨模态异质性导致信息定位困难,二是跨页推理需整合分散证据。为此,我们提出多模态长文档检索增强生成框架MLDocRAG,通过多模态块-查询图(MCQG)将文档内容围绕语义丰富且可答的查询组织起来。MCQG通过多模态文档扩展过程生成细粒度查询,并将其与跨模态、跨页的内容关联。该图结构支持选择性、以查询为中心的检索与结构化证据聚合,显著提升多模态长文档问答的可解释性与连贯性。在MMLongBench-Doc和LongDocURL数据集上的实验表明,MLDocRAG持续提升检索质量与答案准确率,验证了其在多模态长文档理解中的有效性。
原文摘要 · Abstract (English)
Understanding multimodal long-context documents that comprise multimodal chunks such as paragraphs, figures, and tables is challenging due to (1) cross-modal heterogeneity to localize relevant information across modalities, (2) cross-page reasoning to aggregate dispersed evidence across pages. To address these challenges, we are motivated to adopt a query-centric formulation that projects cross-modal and cross-page information into a unified query representation space, with queries acting as abstract semantic surrogates for heterogeneous multimodal content. In this paper, we propose a Multimodal Long-Context Document Retrieval Augmented Generation (MLDocRAG) framework that leverages a Multimodal Chunk-Query Graph (MCQG) to organize multimodal document content around semantically rich, answerable queries. MCQG is constructed via a multimodal document expansion process that generates fine-grained queries from heterogeneous document chunks and links them to their corresponding content across modalities and pages. This graph-based structure enables selective, query-centric retrieval and structured evidence aggregation, thereby enhancing grounding and coherence in multimodal long-context question answering. Experiments on datasets MMLongBench-Doc and LongDocURL demonstrate that MLDocRAG consistently improves retrieval quality and answer accuracy, demonstrating its effectiveness for multimodal long-context understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。