用多跳图文知识图谱增强多模态问答，让大模型回答更准确。
M$^3$KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
- 构建多跳图文知识图谱，支持跨模态实体关联检索
- 引入精准定位与冗余过滤机制，提升答案相关性
- 适合需要强推理与事实一致性的多模态任务
检索增强生成（RAG）已拓展至多模态场景，将多模态大语言模型（MLLMs）与外部知识库如多模态知识图谱（MMKGs）结合。然而，音视频领域多模态RAG仍面临两大挑战：1）现有MMKG的模态覆盖有限且缺乏多跳连接；2）仅依赖共享多模态嵌入空间中的相似度检索，无法过滤无关或冗余知识。为此，我们提出M$^3$KG-RAG，一种多跳多模态知识图谱增强的RAG框架，通过从MMKG中检索与查询对齐的音视频知识，提升MLLM的推理深度与答案忠实度。具体地，设计轻量级多智能体管道构建多跳MMKG（M$^3$KG），包含上下文丰富、跨模态实体的三元组，实现基于输入查询的模态特异性检索。此外，提出GRASP（Grounded Retrieval And Selective Pruning），确保实体与查询精准对齐，评估答案支持相关性，并剪枝冗余上下文，仅保留生成响应所需知识。在多个多模态基准上的实验表明，M$^3$KG-RAG显著优于现有方法，在多模态推理与定位能力上均有提升。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has recently been extended to multimodal settings, connecting multimodal large language models (MLLMs) with vast corpora of external knowledge such as multimodal knowledge graphs (MMKGs). Despite their recent success, multimodal RAG in the audio-visual domain remains challenging due to 1) limited modality coverage and multi-hop connectivity of existing MMKGs, and 2) retrieval based solely on similarity in a shared multimodal embedding space, which fails to filter out off-topic or redundant knowledge. To address these limitations, we propose M$^3$KG-RAG, a Multi-hop Multimodal Knowledge Graph-enhanced RAG that retrieves query-aligned audio-visual knowledge from MMKGs, improving reasoning depth and answer faithfulness in MLLMs. Specifically, we devise a lightweight multi-agent pipeline to construct multi-hop MMKG (M$^3$KG), which contains context-enriched triplets of multimodal entities, enabling modality-wise retrieval based on input queries. Furthermore, we introduce GRASP (Grounded Retrieval And Selective Pruning), which ensures precise entity grounding to the query, evaluates answer-supporting relevance, and prunes redundant context to retain only knowledge essential for response generation. Extensive experiments across diverse multimodal benchmarks demonstrate that M$^3$KG-RAG significantly enhances MLLMs' multimodal reasoning and grounding over existing approaches. Project website: https://kuai-lab.github.io/cvpr2026m3kgrag/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。