让RAG系统评估从模糊打分变为可操作的改进指南
RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines
- 构建六维诊断框架，定位RAG管道中各环节问题
- 通过大模型推理生成自然语言故障解释与优化建议
- 在五个数据集上单次人工调整即显著提升性能
检索增强生成（RAG）系统将大语言模型与外部知识结合，但现有评估方法仅提供汇总分数，无法揭示性能不佳的具体原因。我们提出RAGXplain，一个将性能指标转化为可操作建议的评估框架。该框架围绕‘指标钻石’结构，通过六个诊断维度连接用户输入、检索内容、生成答案及（如有）真实答案。利用大模型推理生成自然语言形式的故障模式解释和优先级干预建议。在五个问答基准上，仅需一次人工引导即可通过实施建议显著提升RAG系统的多项性能指标。我们已开源RAGXplain，以支持可复现性和社区应用。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems couple large language models with external knowledge, yet most evaluation methods report aggregate scores that reveal whether a pipeline underperforms but not where or why. We introduce RAGXplain, an evaluation framework that translates performance metrics into actionable guidance. RAGXplain structures evaluation around a 'Metric Diamond' connecting user input, retrieved context, generated answer, and (when available) ground truth via six diagnostic dimensions. It uses LLM reasoning to produce natural-language failure-mode explanations and prioritized interventions. Across five QA benchmarks, applying RAGXplain's recommendations in a single human-guided pass consistently improves RAG pipeline performance across multiple metrics. We release RAGXplain as open source to support reproducibility and community adoption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。