构建多模态RAG评估基准，支持文本、表格和知识图谱的精准评测。
mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs
- 设计模块化评测框架，拆解检索与生成各环节性能。
- 整合六大数据集，统一转化为可检索文档，覆盖三类数据形式。
- 提供细粒度标注，支持对检索准确率和查询路由的独立评估。
检索增强生成（RAG）已成为提升大语言模型能力的重要范式。然而，现有RAG评估主要聚焦文本检索，且依赖模糊的端到端生成结果评价。为解决这一问题，我们提出mmRAG，一个面向多模态RAG系统的模块化评测基准。该基准整合了来自六个不同问答数据集的查询，涵盖文本、表格和知识图谱，并将其统一转换为可检索文档。为实现对RAG各组件——如检索准确率和查询路由——的直接、细粒度评估，我们采用标准信息检索流程标注文档相关性，并推导数据集相关性。通过在mmRAG上评估多种RAG实现，建立基线性能。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing the capabilities of large language models. However, existing RAG evaluation predominantly focuses on text retrieval and relies on opaque, end-to-end assessments of generated outputs. To address these limitations, we introduce mmRAG, a modular benchmark designed for evaluating multi-modal RAG systems. Our benchmark integrates queries from six diverse question-answering datasets spanning text, tables, and knowledge graphs, which we uniformly convert into retrievable documents. To enable direct, granular evaluation of individual RAG components -- such as the accuracy of retrieval and query routing -- beyond end-to-end generation quality, we follow standard information retrieval procedures to annotate document relevance and derive dataset relevance. We establish baseline performance by evaluating a wide range of RAG implementations on mmRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。