构建复杂临床推理基准,测试医疗智能体真实场景下的分析能力
EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

- 基于MIMIC-IV构建交互式医疗数据库推理任务,支持多表联查与复合推理
- 平均每个查询含31.93个SQL结构组件,顶尖模型准确率仅62.3%
- 揭示医疗大模型在逻辑错误、编码查找和语义理解上的三大缺陷
临床智能体有望提升电子健康记录(EHR)的可及性,但现有基准未能反映真实EHR分析的复杂性,常基于理想化、干净的EHR数据,通过静态SQL生成而非交互式执行。本文提出EHR-Complex,一个大规模交互式临床数据库推理基准。基于MIMIC-IV数据集(365,000名患者,31张表,超过5亿条记录),EHR-Complex包含约5.2万项任务,涵盖六类临床意图,支持个体患者与群体层面查询。每项任务要求智能体在沙盒环境中通过执行SQL或Python代码进行交互。该基准考虑真实世界中纵向多表聚合与组合推理带来的复杂性,平均每个查询包含31.93个SQL结构组件。评估结果表明,这些任务具有极高临床难度,表现最佳模型的精确匹配准确率仅为62.3%。在k=4时,几乎所有模型的Pass^k一致性低于50%,暴露广泛随机脆弱性。对超过3,800条失败轨迹的细粒度分析揭示三大主要失败模式:SQL逻辑错误、医学编码查找失败、语义理解偏差。EHR-Complex为临床智能体提供严格测试平台,并揭示了大规模EHR分析中鲁棒推理仍存在的显著差距。
原文摘要 · Abstract (English)
Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。