让多模态大模型自找相关页面，高效理解长文档
SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding
- 用多模态大模型自身做跨页检索，无需外部解析器
- 在多个公开数据集上达到当前最佳性能
- 适合需要处理长篇图文文档的智能问答场景
多模态大语言模型（MLLM）在图文丰富内容的理解上取得显著进展，但仍难以应对复杂、跨页的视觉化长文档。传统方法依赖文档解析器进行检索增强生成，存在性能与效率瓶颈；而直接将全部页面输入模型则导致计算低效，尤其在长文档场景下。本文提出新型框架SV-RAG（Self-Visual Retrieval-Augmented Generation），使任意MLLM具备长文档理解能力。我们证明，MLLM本身可作为有效的多模态检索器，自动筛选出与问题相关的页面，再基于这些页面作答。SV-RAG通过两个特定的MLLM适配器实现：一个用于证据页检索，另一个用于问答生成。实验结果表明，在多个公开基准上均达到领先水平，验证了该方法的有效性。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have recently shown great progress in text-rich image understanding, yet they still struggle with complex, multi-page visually-rich documents. Traditional methods using document parsers for retrieval-augmented generation suffer from performance and efficiency limitations, while directly presenting all pages to MLLMs leads to inefficiencies, especially with lengthy ones. In this work, we present a novel framework named **S**elf-**V**isual **R**etrieval-**A**ugmented **G**eneration (SV-RAG), which can broaden horizons of any MLLM to support long-document understanding. We demonstrate that **MLLMs themselves can be an effective multimodal retriever** to fetch relevant pages and then answer user questions based on these pages. SV-RAG is implemented with two specific MLLM adapters, one for evidence page retrieval and the other for question answering. Empirical results show state-of-the-art performance on public benchmarks, demonstrating the effectiveness of SV-RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。