构建真实医疗场景下的长期决策评估基准,测试AI对病程演进的理解能力。
LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making

- 基于MIMIC-IV数据构建时序事件流与长上下文记忆数据集
- 平均每患者19.72次住院、每趟44.91个医疗事件,支持多轮交互评估
- 揭示大模型在时间推断和长期决策中的短板,适合临床AI研究者参考
本文提出LongMedBench,一个基于真实电子健康记录(EHR)的长期临床决策评估基准。现有LLM医疗代理评估多集中于短上下文问答与工具使用,但真实医疗具有显著纵向特性,需整合多次就诊、检查及治疗演变的信息。为此,我们通过可复现流程,将MIMIC-IV入院记录与临床文本转化为时序事件流与长上下文记忆数据集,支持代理与临床环境的多轮、长期交互。该基准包含335名患者,平均每人19.72次住院,每次住院44.91个医疗事件。基于长期决策流程,我们设计三类评估任务:事实问答、时间推理与长期决策。实验表明,尽管当前大模型能有效利用显式时间戳,但在隐式时间推断上仍存挑战;检索增强生成(RAG)与代理记忆系统虽提升信息检索表现,但决策任务性能高度依赖模型即时上下文。
原文摘要 · Abstract (English)
In this work, we introduce LongMedBench, a real-world EHR-based benchmark for long-horizon clinical decision-making. Prior evaluations of LLM-based medical agents have largely emphasized short-context knowledge QA and tool use. However, real-world medical care is inherently longitudinal, and clinicians must aggregate evidence across repeated visits, tests, and evolving treatments. Therefore, long-horizon interaction is essential for realistic assessment. LongMedBench is constructed via a reproducible pipeline that integrates MIMIC-IV admission records and clinical notes into time-series event streams and long-context memory datasets, enabling long-horizon, multi-session interactions between agents and a clinical environment. It comprises 335 patients, with 19.72 inpatient visits per patient on average and 44.91 medical events per visit. Guided by the long-horizon decision process, we propose an evaluation taxonomy with three suites: fact-based QA, temporal reasoning, and long-horizon decision-making. This taxonomy measures how agents understand and leverage historical patient information over extended horizons. Our experiments show that while recent LLMs can make good use of explicit timestamps, they have challenges in implicit time inference; The RAG and agent memory system can improve the performance of information retrieval tasks, but the performance of decision-making tasks is highly dependent on the model's immediate context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。