arXiv:2605.22829cs.IRcs.AI2026-05

将文档检索从页面级提升到块级,让大模型更精准理解图文混排文档。

LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding

论文配图:LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding
图 1 · 摘自论文原文
  • 按布局分割文档为语义块,实现细粒度检索
  • 检索准确率提升7.20%,生成时消耗令牌减少73.07%
  • 专为多模态文档设计,适合需要精准理解的场景

多模态检索增强生成(RAG)已成为增强大语言模型(LLMs)外部知识的有效范式。然而,现有系统主要依赖粗粒度的页面级检索,难以捕捉视觉丰富文档中的细粒度语义与版式结构,导致检索精度下降且下游任务中出现冗余内容。为此,我们提出面向版式的细粒度检索增强生成框架(LFRAG),将多模态RAG从页面级推进至块级检索。通过布局分割构建语义连贯的细粒度检索单元,并设计语义-版式融合编码器,利用交叉注意力融合局部语义与全局上下文。结合块级后期交互检索,实现查询与内容的精确对齐,减少无关信息输入。为支持严格评估,我们构建了LFDocQA——一个大规模基准数据集,包含跨多种文档类型的块级标注,用于更精细地评测多模态文档检索与问答任务。在LFDocQA上的实验表明,LFRAG在检索任务上达到当前最优性能,问答准确率优于最佳基线7.20%,生成任务中令牌消耗降低73.07%,验证了其在视觉丰富文档上的高精度与高效率。代码与数据集即将发布。

原文摘要 · Abstract (English)

Multimodal Retrieval-Augmented Generation (RAG) has emerged as an effective paradigm for enhancing Large Language Models (LLMs) with external knowledge. However, existing multimodal RAG systems predominantly rely on coarse-grained page-level retrieval, which fails to capture fine-grained semantic and layout structures in visually rich documents, thereby compromising retrieval accuracy and leading to redundant context in downstream tasks. To address these issues, we propose Layout-oriented Fine-grained Retrieval-Augmented Generation (LFRAG), a novel framework that advances multimodal RAG from page-level to block-level retrieval. We perform layout segmentation to construct semantically coherent fine-grained retrieval units and design a semantic-layout fusion encoder that integrates local semantics with global context via cross-attention. With block-level late interaction retrieval, LFRAG enables precise query-content alignment and reduces irrelevant content for downstream generation. To enable rigorous evaluation, we construct LFDocQA, a large-scale benchmark with block-level annotations spanning diverse document types, designed to assess both multimodal document retrieval and question answering with greater granularity than existing datasets. Extensive experiments on LFDocQA demonstrate that LFRAG achieves state-of-the-art performance on retrieval tasks, outperforms the best baseline by 7.20% in answer accuracy, and reduces token consumption by 73.07% in generation tasks, confirming LFRAG as an accurate and efficient framework for multimodal RAG over visually rich documents. Our code and datasets will be released soon.

文档理解多模态检索增强细粒度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。