揭露医学大模型诊断中的思维定式,提出新评测基准与应对方法
MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosis
- 构建5383对反事实病例,检测模型是否因依赖统计规律误诊
- 顶尖模型在常规病例上准确率超90%,但陷阱病例误诊率高达67%
- 提出双路径推理+动态知识演化框架,提升真实临床决策能力
尽管在医疗基准测试中表现优异,医学大模型在临床诊断中仍存在‘顿悟效应’——过度依赖统计捷径而非患者特异性证据,导致非典型病例误诊。现有评测难以发现这一关键缺陷。我们提出MedEinst,一个包含5,383对临床病例的反事实基准,覆盖49种疾病。每对案例包含一个对照案例和一个‘陷阱’案例,其关键鉴别特征被改变,使诊断结果反转。通过‘偏差陷阱率’(即在正确诊断对照案例的前提下,错误诊断陷阱案例的概率)衡量模型敏感性。对17个大模型的评估显示,前沿模型虽在基准上准确率超90%,但偏差陷阱率普遍超过67%。为此,我们提出ECR-Agent,通过两个组件实现与循证医学标准对齐:(1) 动态因果推理(DCI)采用双路径感知机制,在关联、干预、反事实三个层次进行结构化推理,并执行证据审计;(2) 批判驱动的图与记忆演化(CGME)通过存储验证过的推理路径并动态更新疾病知识图谱,实现系统持续进化。源代码将公开。
原文摘要 · Abstract (English)
Despite achieving high accuracy on medical benchmarks, LLMs exhibit the Einstellung Effect in clinical diagnosis--relying on statistical shortcuts rather than patient-specific evidence, causing misdiagnosis in atypical cases. Existing benchmarks fail to detect this critical failure mode. We introduce MedEinst, a counterfactual benchmark with 5,383 paired clinical cases across 49 diseases. Each pair contains a control case and a "trap" case with altered discriminative evidence that flips the diagnosis. We measure susceptibility via Bias Trap Rate--probability of misdiagnosing traps despite correctly diagnosing controls. Extensive Evaluation of 17 LLMs shows frontier models achieve high baseline accuracy but severe bias trap rates. Thus, we propose ECR-Agent, aligning LLM reasoning with Evidence-Based Medicine standard via two components: (1) Dynamic Causal Inference (DCI) performs structured reasoning through dual-pathway perception, dynamic causal graph reasoning across three levels (association, intervention, counterfactual), and evidence audit for final diagnosis; (2) Critic-Driven Graph and Memory Evolution (CGME) iteratively refines the system by storing validated reasoning paths in an exemplar base and consolidating disease-specific knowledge into evolving illness graphs. Source code is to be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。