用细分问题覆盖度评估RAG系统，发现主流模型仍漏掉一半关键信息。
Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage
- 将问题拆解为核心、背景、追问三类子问题，量化覆盖程度
- 三大商用引擎平均漏掉50%核心子问题，仍有显著提升空间
- 基于核心子问题优化后，生成质量胜过基线74%以上
评估检索增强生成（RAG）系统仍具挑战性，尤其针对无明确答案的开放性问题，需涵盖多个子主题。本文提出一种基于子问题覆盖度的新评估框架，将问题分解为三类：核心、背景与追问，反映其角色与重要性。基于此分类，设计细粒度评估协议，分析包括You.com、Perplexity AI和Bing Chat在内的三个商用生成式问答引擎。结果显示，尽管所有引擎对核心子问题的覆盖率高于背景或追问类，但仍平均遗漏约50%的核心子问题，暴露明显改进空间。此外，子问题覆盖度指标在响应排序中表现优异，准确率达82%，接近人工偏好标注。最后，利用核心子问题优化检索与生成，使新系统在对比基线时获得74%的胜率提升。
原文摘要 · Abstract (English)
Evaluating retrieval-augmented generation (RAG) systems remains challenging, particularly for open-ended questions that lack definitive answers and require coverage of multiple sub-topics. In this paper, we introduce a novel evaluation framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We propose decomposing questions into sub-questions and classifying them into three types -- core, background, and follow-up -- to reflect their roles and importance. Using this categorization, we introduce a fine-grained evaluation protocol that provides insights into the retrieval and generation characteristics of RAG systems, including three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. Interestingly, we find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions, revealing clear opportunities for improvement. Further, sub-question coverage metrics prove effective for ranking responses, achieving 82% accuracy compared to human preference annotations. Lastly, we also demonstrate that leveraging core sub-questions enhances both retrieval and answer generation in a RAG system, resulting in a 74% win rate over the baseline that lacks sub-questions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。