让AI理解图文混排文档，通过多模态知识图谱提升问答准确率。
Multimodal RAG for Unstructured Data:Leveraging Modality-Aware Knowledge Graphs with Hybrid Retrieval
- 构建跨模态知识图谱，融合向量检索与结构化路径搜索。
- 在多个数据集上达到ROUGE-L 0.486，实现完整模态覆盖。
- 适合处理图表、公式等复杂非结构化文档的场景。
当前检索增强生成（RAG）系统主要针对单一文本数据，难以有效处理包含文字、图像、表格、公式和图表的非结构化多模态文档。本文提出一种面向多模态问答的模态感知混合检索架构（MAHA），通过模态感知知识图谱整合密集向量检索与结构化图遍历，编码跨模态语义与关联关系。该设计实现了跨多种模态的语义丰富且上下文敏感的检索。在多个基准数据集上的评估显示，MAHA显著优于基线方法，取得ROUGE-L分数0.486，并实现完整的模态覆盖。结果表明，该方法能有效结合嵌入表示与显式文档结构，支持对非结构化多模态数据的模态感知推理。本工作建立了一个可扩展且可解释的检索框架，推动了RAG系统在多模态场景下的发展。
原文摘要 · Abstract (English)
Current Retrieval-Augmented Generation (RAG) systems primarily operate on unimodal textual data, limiting their effectiveness on unstructured multimodal documents. Such documents often combine text, images, tables, equations, and graphs, each contributing unique information. In this work, we present a Modality-Aware Hybrid retrieval Architecture (MAHA), designed specifically for multimodal question answering with reasoning through a modality-aware knowledge graph. MAHA integrates dense vector retrieval with structured graph traversal, where the knowledge graph encodes cross-modal semantics and relationships. This design enables both semantically rich and context-aware retrieval across diverse modalities. Evaluations on multiple benchmark datasets demonstrate that MAHA substantially outperforms baseline methods, achieving a ROUGE-L score of 0.486, providing complete modality coverage. These results highlight MAHA's ability to combine embeddings with explicit document structure, enabling effective multimodal retrieval. Our work establishes a scalable and interpretable retrieval framework that advances RAG systems by enabling modality-aware reasoning over unstructured multimodal data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。