构建可解释的跨模态知识图谱，让大模型更懂图文关联。
MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs
- 用场景图表示视觉内容，通过谱聚类实现图文实体对齐
- 在CMEL数据集上实体链接准确率提升，复杂推理任务表现更强
- 适合需要精准图文理解与可解释推理的研究者
大型语言模型因静态参数化知识易产生幻觉。检索增强生成（RAG）和GraphRAG通过引入外部知识与知识图谱结构化推理缓解此问题。然而现有方法仍以文本为中心，构建具备显式跨模态语义的细粒度多模态知识图谱（MMKG）仍具挑战。本文提出MMGraphRAG框架，构建可解释的多模态知识图谱，统一文本与视觉知识。该方法将视觉内容表示为结构化场景图，并通过创新的跨模态实体链接方法SpecLink，利用谱聚类联合建模语义相似性与图结构，实现跨模态实体与关系的精准对齐。该设计保留了跨模态的显式实体、关系与推理路径，支持结构感知的检索与生成。为评估性能，我们构建了CMEL数据集，用于细粒度跨模态实体对齐。在CMEL上的实验显示实体链接准确率提升；在DocBench与MMLongBench上的评估表明，MMGraphRAG在复杂多模态推理场景中表现更优，且具备更强鲁棒性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) suffer from hallucinations due to their static parametric knowledge. Retrieval-Augmented Generation (RAG) and GraphRAG mitigate this issue by incorporating external knowledge and structured reasoning over knowledge graphs (KGs). However, existing approaches remain largely text-centric, as constructing fine-grained multimodal knowledge graphs (MMKGs) with explicit cross-modal semantics remains challenging. In this paper, we propose MMGraphRAG, a framework for building interpretable MMKGs that unify textual and visual knowledge. Our approach represents visual content as structured scene graphs and integrates them with textual KGs through a novel cross-modal entity linking method, SpecLink, which leverages spectral clustering to jointly model semantic similarity and graph structure. This design preserves explicit entities, relations, and reasoning paths across modalities, enabling structure-aware retrieval and generation. To support evaluation, we introduce the CMEL dataset, a benchmark for fine-grained cross-modal entity alignment. Experimental results on CMEL demonstrate improved entity linking accuracy, while evaluations on DocBench and MMLongBench show that MMGraphRAG achieves superior performance and stronger robustness, particularly in complex multimodal reasoning scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。