用大模型零样本评估RAG系统,全面又高效。
CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation
- 用一个预训练大模型当裁判,端到端评估五项指标。
- 在BioASQ数据集上区分出Mistral-7B优于Llama系列。
- 比复杂框架更快,适合快速迭代优化RAG系统。
RAG系统通过引入外部知识增强大模型,对需要事实准确性和实时信息的领域至关重要。然而,评估RAG输出在上下文连贯性、问题相关性、事实正确性和信息完整性等方面的综合质量存在挑战。现有方法多依赖简单的词汇重叠度量,无法捕捉细微差别,或采用复杂的多阶段流程,需中间步骤如主张提取或微调专用裁判模型,影响实际效率。为此,我们提出CCR S(上下文连贯性与相关性评分),一套利用单一强大预训练大模型作为零样本、端到端裁判的五项新指标:上下文连贯性(CC)、问题相关性(QR)、信息密度(ID)、答案正确性(AC)和信息召回率(IR)。我们在具有挑战性的BioASQ数据集上评估了六种不同RAG系统配置。分析表明,CCR S能有效区分系统性能,例如证实Mistral-7B阅读器优于Llama系列。我们详细分析了各项指标的分布、收敛/判别效度、平局率、总体统计及区分能力。相比复杂框架RAGChecker,CCR S在召回率和忠实性等关键方面具备相当或更优的区分力,同时计算效率显著更高。因此,CCR S提供了一套实用、全面且高效的RAG系统评估与迭代改进框架。
原文摘要 · Abstract (English)
RAG systems enhance LLMs by incorporating external knowledge, which is crucial for domains that demand factual accuracy and up-to-date information. However, evaluating the multifaceted quality of RAG outputs, spanning aspects such as contextual coherence, query relevance, factual correctness, and informational completeness, poses significant challenges. Existing evaluation methods often rely on simple lexical overlap metrics, which are inadequate for capturing these nuances, or involve complex multi-stage pipelines with intermediate steps like claim extraction or require finetuning specialized judge models, hindering practical efficiency. To address these limitations, we propose CCRS (Contextual Coherence and Relevance Score), a novel suite of five metrics that utilizes a single, powerful, pretrained LLM as a zero-shot, end-to-end judge. CCRS evaluates: Contextual Coherence (CC), Question Relevance (QR), Information Density (ID), Answer Correctness (AC), and Information Recall (IR). We apply CCRS to evaluate six diverse RAG system configurations on the challenging BioASQ dataset. Our analysis demonstrates that CCRS effectively discriminates between system performances, confirming, for instance, that the Mistral-7B reader outperforms Llama variants. We provide a detailed analysis of CCRS metric properties, including score distributions, convergent/discriminant validity, tie rates, population statistics, and discriminative power. Compared to the complex RAGChecker framework, CCRS offers comparable or superior discriminative power for key aspects like recall and faithfulness, while being significantly more computationally efficient. CCRS thus provides a practical, comprehensive, and efficient framework for evaluating and iteratively improving RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。