arXiv:2505.24073cs.AIcs.CL2025-05被引 5

提出多模态RAG设计空间,提升视觉语言模型的准确性与可信度。

mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

  • 系统分析检索、重排、生成三阶段的多模态RAG配置与策略。
  • 不微调情况下实现平均性能提升5%,显著降低幻觉现象。
  • 适合关注视觉语言模型可解释性与动态知识融合的研究者。

大型视觉语言模型(LVLMs)在视觉问答、视觉定位和复杂推理等多模态任务中取得显著进展,但仍受限于静态训练数据、易产生幻觉以及无法验证最新外部证据,影响其在动态现实应用中的表现。检索增强生成(RAG)通过检索机制让LVLM访问大规模知识库,使生成结果基于事实且上下文相关,从而缓解上述问题。本文首次系统剖析面向LVLM的多模态RAG流程,明确考察:(1)检索阶段的模态配置与检索策略;(2)重排阶段缓解位置偏差、提升检索证据相关性的方法;(3)生成阶段如何最优整合检索结果。最后,探索统一的智能体框架,通过自省机制融合重排与生成,使模型能动态选择相关证据并抑制无关内容。全栈式分析揭示关键洞见,无需微调即实现平均性能提升5%。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have made remarkable strides in multimodal tasks such as visual question answering, visual grounding, and complex reasoning. However, they remain limited by static training data, susceptibility to hallucinations, and inability to verify claims against up-to-date, external evidence, compromising their performance in dynamic real-world applications. Retrieval-Augmented Generation (RAG) offers a practical solution to mitigate these challenges by allowing the LVLMs to access large-scale knowledge databases via retrieval mechanisms, thereby grounding model outputs in factual, contextually relevant information. Here in this paper, we conduct the first systematic dissection of the multimodal RAG pipeline for LVLMs, explicitly investigating (1) the retrieval phase: on the modality configurations and retrieval strategies, (2) the re-ranking stage: on strategies to mitigate positional biases and improve the relevance of retrieved evidence, and (3) the generation phase: we further investigate how to best integrate retrieved candidates into the final generation process. Finally, we extend to explore a unified agentic framework that integrates re-ranking and generation through self-reflection, enabling LVLMs to select relevant evidence and suppress irrelevant context dynamically. Our full-stack exploration of RAG for LVLMs yields substantial insights, resulting in an average performance boost of 5% without any fine-tuning.

多模态RAG视觉语言模型知识增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。