让AI理解长文档时,既看内容又懂逻辑关系,提升问答准确率。
MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval
- 构建页面图谱,通过逻辑推理找相关页,不只靠关键词匹配。
- 在4个数据集上问答准确率提升9.68%,检索精度提高7.44%。
- 无需训练即可用,也支持微调,适合想快速部署的开发者。
文档理解是基础性AI能力,文档问答(DocQA)是核心评估任务。传统方法将文档转为文本输入大语言模型(LLM),但会丢失图表等多模态信息。尽管大视觉语言模型(LVLM)可处理多模态内容,但其输入长度受限,难以处理跨页文档。检索增强生成(RAG)虽能选取相关页,但仅依赖语义相似性,忽视页面间的逻辑关联,影响推理效果。为此,我们提出MoLoRAG,一种面向多模态、多页文档理解的逻辑感知检索框架。通过构建捕捉页面间上下文关系的页面图谱,轻量级视觉语言模型执行图遍历,检索出具有逻辑关联的页面,弥补传统方法遗漏。该方法融合语义与逻辑相关性,实现更精准检索。检索后的前K页可输入任意LVLM进行问答。为提升灵活性,提供两种版本:免训练部署方案与优化逻辑判断的微调版本。在四个DocQA数据集上的实验表明,相比直接使用LVLM,平均准确率提升9.68%;相比基线方法,检索精度提升7.44%。代码与数据集已开源于https://github.com/WxxShirley/MoLoRAG。
原文摘要 · Abstract (English)
Document Understanding is a foundational AI capability with broad applications, and Document Question Answering (DocQA) is a key evaluation task. Traditional methods convert the document into text for processing by Large Language Models (LLMs), but this process strips away critical multi-modal information like figures. While Large Vision-Language Models (LVLMs) address this limitation, their constrained input size makes multi-page document comprehension infeasible. Retrieval-augmented generation (RAG) methods mitigate this by selecting relevant pages, but they rely solely on semantic relevance, ignoring logical connections between pages and the query, which is essential for reasoning. To this end, we propose MoLoRAG, a logic-aware retrieval framework for multi-modal, multi-page document understanding. By constructing a page graph that captures contextual relationships between pages, a lightweight VLM performs graph traversal to retrieve relevant pages, including those with logical connections often overlooked. This approach combines semantic and logical relevance to deliver more accurate retrieval. After retrieval, the top-$K$ pages are fed into arbitrary LVLMs for question answering. To enhance flexibility, MoLoRAG offers two variants: a training-free solution for easy deployment and a fine-tuned version to improve logical relevance checking. Experiments on four DocQA datasets demonstrate average improvements of 9.68% in accuracy over LVLM direct inference and 7.44% in retrieval precision over baselines. Codes and datasets are released at https://github.com/WxxShirley/MoLoRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。