arXiv:2608.25986cs.AI2026-08

用多粒度文本上下文增强多模态知识图谱,提升图文检索生成效果

Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphs

论文配图:Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphs
图 1 · 摘自论文原文
  • 在图像局部与全局范围注入互补文本信息,构建上下文增强的多模态知识图谱
  • 在视觉主导数据集上,显著提升多模态RAG的生成质量与准确性
  • 支持多种现有方法,适用于需要图文联合推理的场景

检索增强生成(RAG)广泛用于缓解大语言模型(LLMs)和多模态大语言模型(MLLMs)的幻觉问题。基于知识图谱(KG)的RAG利用结构化知识为(M)LLMs提供高质量外部信息。近期研究将多模态知识图谱(MMKG)作为GraphRAG的知识库,实现跨模态知识整合,进一步提升性能。然而,现有方法通常独立处理不同模态,导致文本上下文在视觉信息提取和融合阶段使用有限,造成图像与文本间的语义鸿沟,限制性能。为此,本文提出一种新型框架构建上下文增强的多模态知识图谱(CEMMKG),在局部与全局层面为每张图像注入互补文本上下文。局部上下文不仅包含邻近文本,还引入语义相关的句子;全局上下文则提供全文摘要。同时采用多粒度设计,捕捉不同层次的语义信息。在选定的视觉主导数据集上的实验表明,CEMMKG能有效利用上下文信息,显著提升基于MMKG的RAG性能,且在多种方法中均具广泛适用性。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models (LLMs) and multimodal large language models (MLLMs). In particular, knowledge graph (KG)-based RAG leverages structured knowledge to provide (M)LLMs with high-quality external information. Building on these works, recent studies have explored multimodal knowledge graphs (MMKGs) as knowledge bases for GraphRAG. This enables Graph RAG to integrate knowledge across multiple modalities, thereby further enhancing its performance. However, existing MMKG-based RAG methods generally follow a common pipeline in which different modalities are largely processed independently before being fusion. As a result, textual context is only used to a limited extent during visual information extraction and subsequent multimodal knowledge fusion. This brings a semantic gap between images and text which limits the multimodal GraphRAG performance. To address this issue, we propose a novel framework for constructing a Context-Enhanced MMKG (CEMMKG) to better support multimodal GraphRAG. The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes. Local context goes beyond the surrounding text by incorporating sentences that are semantically related to the image, while global context provides a summary of the entire passage. We further introduce a multi-granularity design for the local context, allowing it to capture semantically relevant information at different levels of detail. Extensive experiments on the selected vision-centric dataset validate that CEMMKG is effective in leveraging contextual information to improve MMKG-based RAG performance. Moreover, its effectiveness across different MMKG-based RAG methods demonstrates its broad applicability.

多模态知识图谱RAG上下文增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。