arXiv:2511.10552cs.CL2025-11中稿 · AAAI被引 8

让多模态大模型边读边找重点,提升长文档理解效率。

URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding

  • 利用模型早期层自动筛选相关页面,实现推理中动态检索。
  • 在多个数据集上准确率领先,计算量减少44%至56%。
  • 适合需要高效处理长文档的问答与摘要任务场景。

当前多模态大语言模型在长文档理解上仍面临两大挑战:冗余信息引发的信息干扰,以及基于Transformer架构的二次方计算开销。现有方法主要分为两类:令牌压缩会损失细节,引入外部检索器则增加系统复杂性且难以端到端优化。我们深入分析发现,多模态大模型具备类人‘粗粒度到细粒度’的推理模式:早期Transformer层全局关注文档,深层则聚焦相关证据页。受此启发,我们提出URaG框架,将检索与生成统一于单一多模态大模型中。该框架引入轻量级跨模态检索模块,将早期层转化为高效的证据选择器,识别并保留最相关内容页,剔除无关信息。这使深层网络可集中资源处理关键信息,显著提升准确率与效率。大量实验表明,URaG达到当前最优性能,同时计算开销降低44%-56%。代码已开源:https://github.com/shi-yx/URaG。

原文摘要 · Abstract (English)

Recent multimodal large language models (MLLMs) still struggle with long document understanding due to two fundamental challenges: information interference from abundant irrelevant content, and the quadratic computational cost of Transformer-based architectures. Existing approaches primarily fall into two categories: token compression, which sacrifices fine-grained details; and introducing external retrievers, which increase system complexity and prevent end-to-end optimization. To address these issues, we conduct an in-depth analysis and observe that MLLMs exhibit a human-like coarse-to-fine reasoning pattern: early Transformer layers attend broadly across the document, while deeper layers focus on relevant evidence pages. Motivated by this insight, we posit that the inherent evidence localization capabilities of MLLMs can be explicitly leveraged to perform retrieval during the reasoning process, facilitating efficient long document understanding. To this end, we propose URaG, a simple-yet-effective framework that Unifies Retrieval and Generation within a single MLLM. URaG introduces a lightweight cross-modal retrieval module that converts the early Transformer layers into an efficient evidence selector, identifying and preserving the most relevant pages while discarding irrelevant content. This design enables the deeper layers to concentrate computational resources on pertinent information, improving both accuracy and efficiency. Extensive experiments demonstrate that URaG achieves state-of-the-art performance while reducing computational overhead by 44-56%. The code is available at https://github.com/shi-yx/URaG.

多模态长文档检索生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。