提出无需参考答案的RAG评估框架，自动判断回答是否覆盖问题、可验证性如何。
Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability
- 将问题拆解为子问题，答案拆成原子命题，实现细粒度评估
- 在8个数据集上与人工判断相关性高于现有4种评估方法
- 适合想自动化评估RAG系统真实性的研究人员
检索增强生成通过引入检索证据提升大模型事实准确性，但现有评估框架难以在从封闭式事实查询到开放式解释请求的广泛查询类型中提供一致且细粒度的诊断。我们提出Q-CARE，一种无需参考答案、查询无关的评估框架，通过将查询分解为子查询，将答案分解为原子命题，建立基于查询覆盖率和命题可验证性的统一评估原则，得到覆盖感知的检索器指标（C-Prec@k, C-nDCG@k）和命题级别的生成器指标（完整度、简洁度、可验证性）。在涵盖八个数据集的人工标注基准上，Q-CARE与人工判断的相关性高于四种现有RAG评估指标，包括RAGEval和RAGChecker，证明其作为可靠自动化评估框架的有效性。代码与数据公开于https://github.com/DISL-Lab/Q-CaRE-COLM-26。
原文摘要 · Abstract (English)
Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests. We propose Q-CARE, a query-agnostic and fully reference-free framework that enables fine-grained assessment by decomposing queries into sub-queries and answers into atomic claims. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework. Code and data are publicly available at https://github.com/DISL-Lab/Q-CaRE-COLM-26.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。