诊断医疗问答中检索与生成的错误根源，提升AI系统可信度。
RAG-X: Systematic Diagnosis of Retrieval-Augmented Generation for Medical Question Answering
- 分离评估检索和生成模块，精准定位故障环节。
- 发现14%准确率与真实依据间的差距，揭露'虚假准确'现象。
- 适合医疗AI开发者用于构建可验证的临床问答系统。
自动化问答系统越来越多依赖检索增强生成（RAG）技术，将大语言模型（LLMs）锚定在权威医学知识上，以确保人工智能在医疗应用中的临床准确性与患者安全。尽管RAG评估取得进展，现有基准仅关注简单的多选题任务，且评价指标无法有效捕捉复杂问答所需的语义精确性。这些方法无法区分错误源于检索失败还是生成缺陷，限制了开发者的针对性优化。为此，我们提出RAG-X诊断框架，通过信息提取、简答生成和多选题回答三类任务，独立评估检索器与生成器性能。RAG-X引入上下文利用率效率（CUE）指标，将系统表现分解为可解释的象限，分离出真实知识支撑与虚假准确。实验揭示‘准确率谬误’：感知成功率与基于证据的准确率之间存在14%的差距。RAG-X通过暴露隐藏的失败模式，为安全可验证的临床RAG系统提供了诊断透明性。
原文摘要 · Abstract (English)
Automated question-answering (QA) systems increasingly rely on retrieval-augmented generation (RAG) to ground large language models (LLMs) in authoritative medical knowledge, ensuring clinical accuracy and patient safety in Artificial Intelligence (AI) applications for healthcare. Despite progress in RAG evaluation, current benchmarks focus only on simple multiple-choice QA tasks and employ metrics that poorly capture the semantic precision required for complex QA tasks. These approaches fail to diagnose whether an error stems from faulty retrieval or flawed generation, limiting developers from performing targeted improvement. To address this gap, we propose RAG-X, a diagnostic framework that evaluates the retriever and generator independently across a triad of QA tasks: information extraction, short-answer generation, and multiple-choice question (MCQ) answering. RAG-X introduces Context Utilization Efficiency (CUE) metrics to disaggregate system success into interpretable quadrants, isolating verified grounding from deceptive accuracy. Our experiments reveal an ``Accuracy Fallacy", where a 14\% gap separates perceived system success from evidence-based grounding. By surfacing hidden failure modes, RAG-X offers the diagnostic transparency needed for safe and verifiable clinical RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。