arXiv:2608.29582cs.CLcs.AI2026-08

构建医疗诊断多任务基准,测试大模型在矛盾症状下的推理能力

SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence

论文配图:SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence
图 1 · 摘自论文原文
  • 设计三类任务:基础评估、症状相似的多病因辨析、不同症状指向同一疾病
  • 大模型在复杂诊断任务中表现显著下降,依赖统计规律而非真实因果推理
  • 揭示模型对健康状态的保守倾向,可能造成漏诊,适合医学AI安全研究者使用

当前大语言模型(LLMs)评估主要聚焦于事实知识检索,忽视了临床指标与诊断之间复杂、非双射映射的核心挑战。现有基准无法评估模型在诊断歧义场景(相同症状对应不同病因)和诊断收敛场景(不同症状指向同一种疾病)中的真正推理能力。为此,我们提出SUP-MIMIC,一个基于MIMIC-IV-v3.1的多任务框架,包含基础评估(BA)、诊断分歧任务(DDT)和诊断收敛任务(DCT)。DDT用于评估模型在表型相似病例中“一因多果”的辨析能力,DCT则衡量模型识别跨病理通路“多因一果”诊断模式的能力。对先进LLMs的全面评估显示,其在DDT和DCT上的表现相比基础任务显著下降,暴露了模型对统计捷径的系统性依赖,而非真正的因果推理。研究还发现模型存在对‘健康’预测的保守偏倚,暗示在真实医疗环境中存在漏诊的非微小风险。该工作建立了一种量化临床推理鲁棒性的严谨方法,并为提升语言模型在临床医学中的安全性提供了路线图。

原文摘要 · Abstract (English)

Current evaluations of large language models (LLMs) primarily focus on factual knowledge retrieval, overlooking the fundamental challenge of navigating the complex, non-bijective mappings between clinical indicators and diagnoses. Existing benchmarks fail to assess whether large language models truly possess the reasoning capability required for diagnostic ambiguity scenarios, where identical clinical presentations may correspond to different etiologies, and diagnostic convergence scenarios, where heterogeneous symptoms ultimately indicate the same disease. To address this issue, we propose SUP-MIMIC, a multi-task framework utilizing MIMIC-IV-v3.1 that comprises Basic Assessment (BA), Diagnostic Divergence Task (DDT), and Diagnostic Convergence Task (DCT). Specifically, DDT is designed to evaluate the model's "one-to-many" disambiguation capability among phenotypically similar cases, while DCT assesses the model's ability to identify "many-to-one" diagnostic patterns across different pathophysiological pathways. Comprehensive evaluation of state-of-the-art LLMs reveals substantial performance degradation on DDT and DCT compared to baseline tasks, exposing a systemic reliance on statistical shortcuts over genuine causal reasoning. Our findings further highlight a conservative bias toward "healthy" predictions, implying non-trivial risks for missed diagnoses in realistic medical settings. This work establishes a rigorous methodology for quantifying clinical reasoning robustness and provides a roadmap for enhancing the safety of language models in clinical medicine.

医疗AI大模型评测临床推理诊断歧义

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。