首个聚焦医学推理的检索基准,挑战现有系统理解深层诊断逻辑的能力。
R2MED: A Benchmark for Reasoning-Driven Medical Retrieval
- 构建三类临床任务的推理驱动检索数据集,覆盖12个身体系统
- 顶尖模型仅达41.4 nDCG@10,凸显当前技术与临床需求差距
- 适合研究医疗AI推理、临床决策支持系统的开发者使用
当前医学检索基准多依赖词汇或浅层语义匹配,忽视临床决策中关键的推理需求。医生常需检索支持诊断假设的权威证据,而这些证据往往与患者症状表面描述重合度低。为此,我们提出R2MED,首个专为推理驱动医学检索设计的基准,包含876个查询,涵盖问答参考检索、临床证据检索和临床病例检索三类任务,源自五个代表性医学场景及十二个身体系统,反映真实世界信息需求的复杂性与多样性。我们在R2MED上评估15种主流检索系统,发现最优模型仅获31.4 nDCG@10;经典重排序与生成增强方法提升有限。尽管大型推理模型通过中间推理生成改善性能,最佳结果仍止步于41.4 nDCG@10。这表明现有检索技术与真实临床任务的推理要求之间存在显著差距。我们开放R2MED数据与代码,以推动下一代具备更强推理能力的医学检索系统发展。
原文摘要 · Abstract (English)
Current medical retrieval benchmarks primarily emphasize lexical or shallow semantic similarity, overlooking the reasoning-intensive demands that are central to clinical decision-making. In practice, physicians often retrieve authoritative medical evidence to support diagnostic hypotheses. Such evidence typically aligns with an inferred diagnosis rather than the surface form of a patient's symptoms, leading to low lexical or semantic overlap between queries and relevant documents. To address this gap, we introduce R2MED, the first benchmark explicitly designed for reasoning-driven medical retrieval. It comprises 876 queries spanning three tasks: Q&A reference retrieval, clinical evidence retrieval, and clinical case retrieval. These tasks are drawn from five representative medical scenarios and twelve body systems, capturing the complexity and diversity of real-world medical information needs. We evaluate 15 widely-used retrieval systems on R2MED and find that even the best model achieves only 31.4 nDCG@10, demonstrating the benchmark's difficulty. Classical re-ranking and generation-augmented retrieval methods offer only modest improvements. Although large reasoning models improve performance via intermediate inference generation, the best results still peak at 41.4 nDCG@10. These findings underscore a substantial gap between current retrieval techniques and the reasoning demands of real clinical tasks. We release R2MED as a challenging benchmark to foster the development of next-generation medical retrieval systems with enhanced reasoning capabilities. Data and code are available at https://github.com/R2MED/R2MED
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。