检测医疗问答中不同数据源导致答案差异，揭示现有评估的盲区。
Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG
- 以多机构手册为源，分析同一问题的答案分歧
- 发现检索质量越高，答案差异越显著，超出以往估计
- 提供可复用框架，适用于法律教育等多领域RAG
在多作者机构语料上部署的检索增强生成（RAG）系统，对同一问题可能因检索来源不同而给出不同答案，这是当前单金标准评估范式无法诊断的问题。我们主张源依赖性是缺失的NLP评估维度，审计它需将评价单位从答案正确性转向源间关系。在移植患者教育场景中，机构来源明显存在分歧，为此发布三个成果：TransplantQA，一个包含真实患者问题的基准，每个问题由多个机构手册作为候选源进行答案生成；HERO-QA，一种分层检索策略，用于定位并审计每个答案的来源依据；以及结构化输出评判器，基于经验证的五标签分类体系评分源间关系。大规模测试表明，更好的检索揭示了远超先前估计的分歧程度——低估了其普遍性而非强度。该框架具有领域通用性，可迁移至法律和教育RAG，测量源依赖性是部署多源NLP系统的责任。
原文摘要 · Abstract (English)
A retrieval-augmented generation (RAG) system deployed over a multi-author institutional corpus can give a different answer to the same question depending on which source it retrieves -- a failure mode the dominant single-gold-answer paradigm cannot diagnose. We argue that source-dependence is a missing axis of NLP evaluation, and that auditing it means shifting the unit of evaluation from answer correctness to the inter-source relationship. We make this concrete in transplant patient education, where institutional sources demonstrably disagree, releasing three artefacts: TransplantQA, a benchmark of real patient questions, each answered by grounding generation in multiple institutional handbooks as candidate sources; HERO-QA, a hierarchical retrieval strategy that grounds and audits each answer; and a structured-output judge that scores inter-source relationships on a validated 5-label taxonomy. At scale, better retrieval reveals far more disagreement than prior estimates suggested -- understating its prevalence, not its intensity. The framework is domain-agnostic and transfers to legal and educational RAG: measuring source-dependence is a responsibility for deployed multi-source NLP generally.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。