构建多轮临床诊断数据集，评估大模型真实问诊能力
MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation
- 从三大来源构建多轮诊断对话数据集，覆盖常见与罕见病
- 采用标准化医学知识库编码，支持精准诊断评估
- 引入临床知识对齐指标，超越传统准确率评价
临床诊断本质上是交互式、渐进式的，但当前主流医学大模型评估仍依赖静态问答或模板化对话，难以反映模型在动态诊疗场景中的表现，尤其在多轮交互中准确率与可靠性显著下降。为解决此问题，我们提出MTDiag，一个大规模多轮诊断对话数据集，源自三个异构来源：DDXPlus、MIMIC-IV及已发表病例报告（AJCR），涵盖急诊常见症状及长尾罕见、非典型疾病。所有病例均基于最全面且广泛使用的医学知识库（UMLS概念标识符，附ICD-10诊断编码）进行规范化处理。我们公开了数据结构规范、基于UserLM-8B的语句生成流程，以及由医师验证的自然语言转换数据集。尤为重要的是，我们提出并论证了以临床知识为基准的评估指标，用于衡量大模型作为诊断代理在多轮鉴别诊断任务中的表现，超越单纯诊断准确率。
原文摘要 · Abstract (English)
Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues. These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings. To address this issue, we present MTDiag, a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous sources: DDXPlus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions. All cases are normalized into a canonical schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes). We release the schema, a UserLM-8B-based utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances. Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。