诊断视觉增强生成的评估平台，发现复杂查询下幻觉严重。
MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation
- 构建多模态评测平台，融合多种基准数据集。
- 困难和模糊查询下准确率显著下降，幻觉问题突出。
- 提供逐条判断诊断工具，适合系统优化者使用。
多模态检索增强生成（Visual RAG）通过整合视觉与文本证据，显著提升了问答能力。然而现有评估未能系统考虑查询难度与歧义性。本文提出MRAG-Suite，一个集成多种多模态基准（WebQA、Chart-RAG、Visual-RAG、MRAG-Bench）的诊断评估平台。引入基于难度和歧义性的过滤策略，并开发了MM-RAGChecker——一项基于断言级别的诊断工具。实验结果表明，在困难与模糊查询下，模型准确率出现明显下降，暴露出普遍的幻觉现象。MM-RAGChecker能有效识别这些问题，为未来Visual RAG系统的改进提供指导。
原文摘要 · Abstract (English)
Multimodal Retrieval-Augmented Generation (Visual RAG) significantly advances question answering by integrating visual and textual evidence. Yet, current evaluations fail to systematically account for query difficulty and ambiguity. We propose MRAG-Suite, a diagnostic evaluation platform integrating diverse multimodal benchmarks (WebQA, Chart-RAG, Visual-RAG, MRAG-Bench). We introduce difficulty-based and ambiguity-aware filtering strategies, alongside MM-RAGChecker, a claim-level diagnostic tool. Our results demonstrate substantial accuracy reductions under difficult and ambiguous queries, highlighting prevalent hallucinations. MM-RAGChecker effectively diagnoses these issues, guiding future improvements in Visual RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。