评测多模态RAG系统思维与观察能力，发现闭源模型显著优于开源。
FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
- 构建93个跨模态问题数据集，评估系统对文本、表格、图像的综合处理能力。
- 提出短语级召回率与嵌入分类器，精准衡量正确性与幻觉检测效果。
- 闭源模型在复杂多文档任务中表现更优，适合关注系统可靠性研究者。
检索增强生成（RAG）已成为提升大语言模型事实准确性的有前景范式。本文提出一个整体评估框架，用于评测RAG管道在摄入、检索和推理多模态信息方面的能力，区别于以往仅关注检索的基准。我们构建了由93个问题组成的小规模人工数据集，涵盖文本、表格、图像及跨文档多模态信息；提出短语级召回率作为正确性度量；设计近邻嵌入分类器以识别潜在幻觉；对比评估了2个开源检索机制构建的管道与4个闭源基础模型；并进行第三方人类评估以验证正确性与幻觉检测指标的对齐性。结果显示，闭源管道在正确性与幻觉指标上均显著优于开源管道，尤其在依赖多模态和跨文档信息的问题上差距更大。人类评估显示，正确性指标平均一致性为4.62（1-5评分），幻觉检测为4.53，表明指标与人类判断高度一致。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) has emerged as a promising paradigm for improving factual accuracy in large language models (LLMs). We introduce a benchmark designed to evaluate RAG pipelines as a whole, evaluating a pipeline's ability to ingest, retrieve, and reason about several modalities of information, differentiating it from existing benchmarks that focus on particular aspects such as retrieval. We present (1) a small, human-created dataset of 93 questions designed to evaluate a pipeline's ability to ingest textual data, tables, images, and data spread across these modalities in one or more documents; (2) a phrase-level recall metric for correctness; (3) a nearest-neighbor embedding classifier to identify potential pipeline hallucinations; (4) a comparative evaluation of 2 pipelines built with open-source retrieval mechanisms and 4 closed-source foundation models; and (5) a third-party human evaluation of the alignment of our correctness and hallucination metrics. We find that closed-source pipelines significantly outperform open-source pipelines in both correctness and hallucination metrics, with wider performance gaps in questions relying on multimodal and cross-document information. Human evaluation of our metrics showed average agreement of 4.62 for correctness and 4.53 for hallucination detection on a 1-5 Likert scale (5 indicating "strongly agree").
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。