将多模态检索证据细化到片段级别,提升生成准确性。
Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAG

- 把文档拆成文本句段和视觉区域,选最相关片段作证据
- 用信息增益指标评估片段价值,减少冗余内容27%以上
- 适合追求高准确率与低上下文长度的多模态生成任务
多模态检索增强生成(MRAG)通过引入外部证据来减少多模态大模型(MLLM)的幻觉。然而现有方法将检索到的证据视为不可分割的整体,假设文档内所有内容都同等有用。实际上,仅一小部分内容与查询相关,其余部分会引入噪声导致性能下降。为此,本文提出碎片级证据选择框架FES-RAG,将多模态文档分解为句子级文本片段和区域级视觉片段,实现对生成支持证据的精准定位。引入片段信息增益(FIG)作为衡量标准,评估每个片段对生成置信度的边际贡献,并基于大模型蒸馏出轻量级选择器,在低推理开销下实现高精度筛选。在M2RAG基准测试中,FES-RAG相较当前最优文档级方法,在CIDEr指标上提升最高达27%,同时显著降低上下文长度,提升事实准确性和生成连贯性。
原文摘要 · Abstract (English)
Multimodal Retrieval-Augmented Generation (MRAG) is widely adopted for Multimodal Large Language Models (MLLMs) with external evidence to reduce hallucinations. Despite its success, most existing MRAG frameworks treat retrieved evidence as indivisible documents, implicitly assuming that all content within a document is equally informative. In practice, however, sometimes only a small fraction of a document is relevant to a given query, while the remaining content introduces substantial noise that may lead to performance degradation. We address this fundamental limitation by reframing MRAG as a fine-grained evidence selection problem. We propose Fragment-level Evidence Selection for RAG (FES-RAG), a framework that selects atomic multimodal fragments rather than entire documents as grounding evidence. FES-RAG decomposes retrieved multimodal documents into sentence-level textual fragments and region-level visual fragments, enabling precise identification of evidence that directly supports generation. To guide fragment selection, we introduce Fragment Information Gain (FIG), a principled metric that measures the marginal contribution of each fragment to the MLLM's generation confidence. Based on FIG, we distill fragment-level utility judgments from a high-capacity MLLM into a lightweight selector, achieving accurate evidence selection with low inference overhead. Experiments on the M2RAG benchmark show that FES-RAG consistently outperforms state-of-the-art document-level MRAG methods, achieving up to 27 percent relative improvement in CIDEr. By selecting fewer yet more informative fragments, our approach substantially reduces context length while improving factual accuracy and generation coherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。