构建首个面向RAG系统的精细化评估基准，支持组件级分析。
MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation
- 设计7560个问答对与37800条检索池数据，支持精准评估。
- 提出噪声敏感、上下文误读等4项新指标，揭示系统脆弱性。
- 适合研究RAG模型对齐与鲁棒性的研究人员使用。
检索增强生成（RAG）通过引入外部知识提升了大语言模型的生成能力，但其评估仍面临挑战，因检索与生成模块间存在复杂交互。现有基准难以实现组件级细粒度评估。本文提出MIRAGE，一个专为RAG评估设计的问答数据集，包含7,560个精心构造的实例，映射至37,800条检索条目，可高效精确评估检索与生成任务。我们还引入新型评估指标，衡量RAG系统的适应性，涵盖噪声脆弱性、上下文可接受性、上下文不敏感性和上下文误读等维度。通过对多种检索器-语言模型组合的全面实验，揭示了最优模型对齐策略及RAG系统内部的细微动态。数据集与评估代码已公开，便于在各类研究场景中集成与定制。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has gained prominence as an effective method for enhancing the generative capabilities of Large Language Models (LLMs) through the incorporation of external knowledge. However, the evaluation of RAG systems remains a challenge, due to the intricate interplay between retrieval and generation components. This limitation has resulted in a scarcity of benchmarks that facilitate a detailed, component-specific assessment. In this work, we present MIRAGE, a Question Answering dataset specifically designed for RAG evaluation. MIRAGE consists of 7,560 curated instances mapped to a retrieval pool of 37,800 entries, enabling an efficient and precise evaluation of both retrieval and generation tasks. We also introduce novel evaluation metrics aimed at measuring RAG adaptability, encompassing dimensions such as noise vulnerability, context acceptability, context insensitivity, and context misinterpretation. Through comprehensive experiments across various retriever-LLM configurations, we provide new insights into the optimal alignment of model pairs and the nuanced dynamics within RAG systems. The dataset and evaluation code are publicly available, allowing for seamless integration and customization in diverse research settings\footnote{The MIRAGE code and data are available at https://github.com/nlpai-lab/MIRAGE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。