arXiv:2604.04969cs.IRcs.AI2026-04被引 2

用多粒度图结构提升多模态生成的准确性和效率

MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

论文配图:MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation
图 1 · 摘自论文原文
  • 构建分层多模态知识图,融合文本与视觉信息为统一节点
  • 支持多跳推理,跨模态检索性能优于现有方法43.3倍加速
  • 适合需要精准推理的多模态应用,如医疗、智能客服

检索增强生成(RAG)可缓解多模态大模型的幻觉问题,但现有系统在复杂跨模态推理上表现不佳。传统向量检索忽略结构依赖,现有图方法依赖昂贵的‘转文本’流程,丢失细粒度视觉信息。为此,我们提出轻量级多粒度图RAG框架MG²-RAG,联合优化图构建、模态融合与跨模态检索。该框架通过轻量级文本解析与实体驱动的视觉定位,构建层次化多模态知识图,将文本实体与视觉区域融合为保留原始证据的统一节点。在此基础上,引入多粒度图检索机制,聚合密集相似性并传播相关性,支持结构化多跳推理。在四个代表性多模态任务(检索、基于知识的VQA、推理、分类)上的实验表明,MG²-RAG持续达到当前最优性能,同时相比先进图框架实现平均43.3倍速度提升和23.9倍成本降低。

原文摘要 · Abstract (English)

Retrieval-Augmented Generation (RAG) mitigates hallucinations in Multimodal Large Language Models (MLLMs), yet existing systems struggle with complex cross-modal reasoning. Flat vector retrieval often ignores structural dependencies, while current graph-based methods rely on costly ``translation-to-text'' pipelines that discard fine-grained visual information. To address these limitations, we propose \textbf{MG$^2$-RAG}, a lightweight \textbf{M}ulti-\textbf{G}ranularity \textbf{G}raph \textbf{RAG} framework that jointly improves graph construction, modality fusion, and cross-modal retrieval. MG$^2$-RAG constructs a hierarchical multimodal knowledge graph by combining lightweight textual parsing with entity-driven visual grounding, enabling textual entities and visual regions to be fused into unified multimodal nodes that preserve atomic evidence. Building on this representation, we introduce a multi-granularity graph retrieval mechanism that aggregates dense similarities and propagates relevance across the graph to support structured multi-hop reasoning. Extensive experiments across four representative multimodal tasks (i.e., retrieval, knowledge-based VQA, reasoning, and classification) demonstrate that MG$^2$-RAG consistently achieves state-of-the-art performance while reducing graph construction overhead with an average 43.3$\times$ speedup and 23.9$\times$ cost reduction compared with advanced graph-based frameworks.

多模态知识图谱RAG推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。