arXiv:2503.06073cs.CLcs.AI2025-03NeurIPS被引 44

GEM让大模型读懂心电图,能精准关联波形与诊断。

GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images

  • 双编码器融合心电时序数据与12导联图像特征
  • 诊断准确率提升7.4%,证据定位能力提高24.8%
  • 适合临床辅助诊断系统开发人员使用

尽管近期多模态大语言模型(MLLMs)在自动化心电图(ECG)解读方面取得进展,但仍面临两大挑战:(1) 时序信号与视觉心电图表示之间多模态协同不足;(2) 诊断与细微波形证据之间的可解释性有限。我们提出GEM,首个统一心电图时序数据、12导联心电图图像与文本的MLLM,实现基于证据的、贴近临床医生思维的心电图解读。GEM通过三项核心创新实现:双编码器分别提取时序与图像特征,跨模态对齐促进多模态理解,知识引导指令生成构建高粒度的接地数据(ECG-Grounding),将诊断结果与可测量参数(如QRS/PR间期)关联。此外,我们提出了‘基于证据的心电图理解’任务,一个以临床为导向的基准测试,全面评估模型在接地理解方面的能力。在现有及自建基准上的实验表明,GEM显著提升了预测性能(CSN提升7.4%)、可解释性(提升22.7%)和接地能力(提升24.8%),更适用于真实临床场景。代码开源:https://github.com/lanxiang1017/GEM.git

原文摘要 · Abstract (English)

While recent multimodal large language models (MLLMs) have advanced automated ECG interpretation, they still face two key limitations: (1) insufficient multimodal synergy between time series signals and visual ECG representations, and (2) limited explainability in linking diagnoses to granular waveform evidence. We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation. GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations: a dual-encoder framework extracting complementary time series and image features, cross-modal alignment for effective multimodal understanding, and knowledge-guided instruction generation for generating high-granularity grounding data (ECG-Grounding) linking diagnoses to measurable parameters ($e.g.$, QRS/PR Intervals). Additionally, we propose the Grounded ECG Understanding task, a clinically motivated benchmark designed to comprehensively assess the MLLM's capability in grounded ECG understanding. Experimental results on both existing and our proposed benchmarks show GEM significantly improves predictive performance (CSN $7.4\% \uparrow$), explainability ($22.7\% \uparrow$), and grounding ($24.8\% \uparrow$), making it more suitable for real-world clinical applications. GitHub repository: https://github.com/lanxiang1017/GEM.git

心电图分析多模态模型医学AI可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。