arXiv:2510.03663cs.CLcs.CV2025-10被引 21

首个真实文档多模态检索增强生成评测基准,解决现有评估碎片化问题。

UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG

  • 从真实PDF提取文本、表格、图表证据,构建多类型问答对。
  • 多模态融合模型表现优于单一模态和联合检索,证明图文协同必要性。
  • 适合研究多模态RAG、文档理解与智能助手的开发者与研究者。

多模态检索增强生成(MM-RAG)是将大语言模型与智能体应用于真实知识库的关键方法,但当前评估体系分散——仅关注文本或图像单独作用,或采用简化多模态设置,无法覆盖以文档为中心的真实多模态场景。本文提出UniDoc-Bench,首个基于k个真实领域PDF页面的大规模、高保真度多模态RAG评测基准。其数据流水线从文本、表格与图表中提取并关联证据,生成涵盖事实检索、比较、摘要与逻辑推理的多模态问答对。所有问答对经多名人工标注者与专家评审验证。该基准支持四种范式:1)纯文本,2)纯图像,3)多模态文本-图像融合,4)多模态联合检索,在统一协议下实现公平对比,包含标准化候选池、提示模板与评估指标。该基准亦可用于视觉问答(VQA)任务。实验表明,多模态融合型RAG系统持续优于单模态及联合嵌入式检索,说明仅依赖文本或图像均不充分,且当前多模态嵌入仍存不足。分析进一步揭示视觉信息如何补充文本证据,识别系统性失败模式,并为构建更鲁棒的多模态RAG流程提供可操作建议。

原文摘要 · Abstract (English)

Multimodal retrieval-augmented Generation (MM-RAG) is a key approach for applying large language models (LLMs) and agents to real-world knowledge bases, yet current evaluations are fragmented -- focusing on either text or images in isolation, or simplified multimodal setup, failing to capture document-centric multimodal use cases. In this paper, we introduce UniDoc-Bench, the first large-scale, realistic benchmark for MM-RAG built from $k$ real-world PDF pages across domains. Our pipeline extracts and links evidence from text, tables, and figures, then generates multimodal QA pairs spanning factual retrieval, comparison, summarization, and logical reasoning queries. To ensure reliability, all of QA pairs are validated by multiple human annotators and expert adjudication. UniDoc-Bench supports apples-to-apples comparison across four paradigms: 1) text-only, 2) image-only, 3) \emph{multimodal} text-image fusion and 4) multimodal joint retrieval -- under a unified protocol with standardized candidate pools, prompts, and evaluation metrics. UniDoc-Bench can also be used to evaluate Visual Question Answering (VQA) tasks. Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval, indicating that neither text nor images alone are sufficient and that current multimodal embeddings remain inadequate. Beyond benchmarking, our analysis reveals when and how visual context complements textual evidence, uncovers systematic failure modes, and offers actionable guidance for developing more robust MM-RAG pipelines.

多模态RAG文档理解评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。