评估RAG系统对无法回答问题的识别能力，发现组件选择影响拒答效果。
Unanswerability Evaluation for Retrieval Augmented Generation
- 构建六类不可回答问题分类体系，自动生成挑战性测试集。
- 揭示不同组件组合在可回答与不可回答问题间的性能权衡。
- 适合想提升RAG系统鲁棒性的研究人员和开发者使用。
现有检索增强生成（RAG）系统的评估框架主要关注可回答查询，却忽视了对不可回答请求的恰当拒绝。本文提出UAEval4RAG框架，用于评估RAG系统处理不可回答查询的能力。我们定义了六类不可回答问题的分类体系，并设计自动化方法为任意知识库生成多样且具有挑战性的未回答查询，引入未回答率和可接受拒答率等指标。在多种RAG组件（包括检索模型、重写方法、重排序器、语言模型及提示策略）上进行实验，揭示了系统性能中的隐藏权衡。研究结果表明，组件选择与提示设计在平衡可回答问题的准确性与不可回答问题的高拒答率方面起关键作用。UAEval4RAG为构建更鲁棒、可靠的RAG系统提供了重要洞察与工具。
原文摘要 · Abstract (English)
Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but they overlook the importance of appropriately rejecting unanswerable requests. In this paper, we introduce UAEval4RAG, a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively. We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries for any given knowledge base with unanswered ratio and acceptable ratio metrics. We conduct experiments with various RAG components, including retrieval models, rewriting methods, rerankers, language models, and prompting strategies, and reveal hidden trade-offs in performance of RAG systems. Our findings highlight the critical role of component selection and prompt design in optimizing RAG systems to balance the accuracy of answerable queries with high rejection rates of unanswerable ones. UAEval4RAG provides valuable insights and tools for developing more robust and reliable RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。