长上下文模型能更好预测电子病历，尤其在复杂病例中表现更稳。
Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs
- 用Mamba等长序列架构建模超1万条医疗事件的完整病历
- 在EHRSHOT上9/14任务超越现有最佳模型，提升显著
- 首次揭示病历重复、时间不均、病情递增对模型的影响
基于电子病历（EHR）训练的基础模型（FMs）在多项临床预测任务中达到顶尖水平，但多数模型上下文窗口小于1000个标记。这限制了其对超过1万条事件的完整患者病历建模。近期子二次长上下文架构（如Mamba）提供了新可能，但其在EHR数据中的应用尚未充分研究。本文首次系统评估上下文长度对建模EHR数据的影响。结果表明，更长上下文模型显著提升预测性能——我们的Mamba模型在EHRSHOT基准上9/14任务超越现有最佳。然而，临床应用不仅需高精度，还需对病历特性鲁棒性。因此我们评估模型在三个此前未被重视的特性上的表现：（1）“复制向前”诊断导致序列中出现人工重复；（2）事件间时间间隔不规则，造成上下文内跨度差异大；（3）疾病随时间复杂度自然上升，使后期记录更难预测。分层分析显示，这些特性水平越高，模型表现越差，但长上下文模型对此类极端情况更具鲁棒性。本工作展示了长上下文架构在建模医疗序列数据中的潜力，并为非自然语言领域提出新挑战。代码与模型已开源。
原文摘要 · Abstract (English)
Foundation Models (FMs) trained on Electronic Health Records (EHRs) have achieved state-of-the-art results on numerous clinical prediction tasks. However, most existing EHR FMs have context windows of <1k tokens. This prevents them from modeling full patient EHRs which can exceed 10k's of events. Recent advancements in subquadratic long-context architectures (e.g., Mamba) offer a promising solution. However, their application to EHR data has not been well-studied. We address this gap by presenting the first systematic evaluation of the effect of context length on modeling EHR data. We find that longer context models improve predictive performance -- our Mamba-based model surpasses the prior state-of-the-art on 9/14 tasks on the EHRSHOT prediction benchmark. For clinical applications, however, model performance alone is insufficient -- robustness to the unique properties of EHR is crucial. Thus, we also evaluate models across three previously underexplored properties of EHR data: (1) the prevalence of "copy-forwarded" diagnoses which creates artificial repetition of tokens within EHR sequences; (2) the irregular time intervals between EHR events which can lead to a wide range of timespans within a context window; and (3) the natural increase in disease complexity over time which makes later tokens in the EHR harder to predict than earlier ones. Stratifying our EHRSHOT results, we find that higher levels of each property correlate negatively with model performance, but that longer context models are more robust to more extreme levels of these properties. Our work highlights the potential for using long-context architectures to model EHR data, and offers a case study for identifying new challenges in modeling sequential data motivated by domains outside of natural language. We release our models and code at: https://github.com/som-shahlab/long_context_clues
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。