构建双维度难度矩阵，精细评估RAG系统多跳问答能力
GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluation
- 用知识图谱与语义聚类生成可控难度的多跳问答数据
- 发现错误率与推理深度和语义距离高度相关
- 适合研究RAG系统在复杂推理中的瓶颈
检索增强生成（RAG）系统广泛应用于知识密集型自然语言任务，但现有评估常忽略真实场景中所需的多步推理与结构复杂性。这些基准未能考虑检索难度与推理深度之间的交互关系。为此，我们提出GRADE框架，从两个正交维度建模任务难度：(1) 推理深度（以推理步数/跳数衡量），(2) 查询与支持证据间的语义距离。我们基于事实新闻文章构建合成多跳问答数据集，通过提取知识图谱并利用语义聚类恢复缺失链接，实现多样化且难度可控的查询生成。框架核心为结合生成端与检索端难度的二维难度矩阵。跨多个领域和模型的实验表明，错误率与我们的难度度量高度相关，验证了其诊断价值。GRADE可实现对RAG性能的细粒度分析，并为实际应用中多跳推理的评估与优化提供可扩展基础。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems are widely adopted in knowledge-intensive NLP tasks, but current evaluations often overlook the structural complexity and multi-step reasoning required in real-world scenarios. These benchmarks overlook key factors such as the interaction between retrieval difficulty and reasoning depth. To address this gap, we propose GRADE, a novel evaluation framework that models task difficulty along two orthogonal dimensions: (1) reasoning depth, defined by the number of inference steps (hops), and (2) semantic distance between the query and its supporting evidence. We construct a synthetic multi-hop QA dataset from factual news articles by extracting knowledge graphs and augmenting them through semantic clustering to recover missing links, allowing us to generate diverse and difficulty-controlled queries. Central to our framework is a 2D difficulty matrix that combines generator-side and retriever-side difficulty. Experiments across multiple domains and models show that error rates strongly correlate with our difficulty measures, validating their diagnostic utility. GRADE enables fine-grained analysis of RAG performance and provides a scalable foundation for evaluating and improving multi-hop reasoning in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。