构建统一心电多模态框架,实现信号与图像联合建模。
HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding
- 设计双流对齐机制,在共享空间中联合建模心电信号与图像。
- 在40万条高质量心电指令数据上训练,提升跨模态理解能力。
- 适合医疗大模型研究者拓展生理信号分析能力。
尽管心电图(ECG)在心血管诊疗中占据主导地位,但其固有的数据形式与表征模式给医学多模态大语言模型(Med-MLLMs)带来跨模态语义对齐挑战。为此,我们提出Heartcare Suite:一个面向双信号-图像建模与理解的统一心电多模态套件。包含三部分:(i) Heartcare-400K——基于自研数据流水线引擎HeartAgent,整合顶级医院高质量临床心电报告与开源数据构建的细粒度心电指令数据集;(ii) Heartcare-Bench——系统性基准测试,评估模型在多视角心电理解与跨模态泛化能力,为优化心电理解模型提供指导;(iii) HeartcareGPT——基于结构感知离散分词器Beat,提出双流投影对齐(DSPA)范式,通过双编码器投影对齐机制,在共享特征空间中联合优化原生心电信号与图像表示。HeartcareGPT在多种心电理解任务中均取得一致提升,验证了统一建模范式的有效性与高质量数据管道的必要性,为将Med-MLLMs拓展至生理信号领域奠定方法基础。项目已开源:https://github.com/ZJU4HealthCare/HeartcareGPT。
原文摘要 · Abstract (English)
Although electrocardiograms (ECG) play a dominant role in cardiovascular diagnosis and treatment, their intrinsic data forms and representational patterns pose significant challenges for medical multimodal large language models (Med-MLLMs) in achieving cross-modal semantic alignment. To address this gap, we propose Heartcare Suite, a unified ECG suite designed for dual signal-image modeling and understanding: (i) Heartcare-400K. A fine-grained ECG instruction dataset on top of our data pipeline engine--HeartAgent--by integrating high quality clinical ECG reports from top hospitals with open-source data. (ii) Heartcare-Bench. A systematic benchmark assessing performance of models in multi-perspective ECG understanding and cross-modal generalization, providing guidance for optimizing ECG comprehension models. (iii) HeartcareGPT. Built upon a structure-aware discrete tokenizer Beat, we propose Dual Stream Projection Alignment (DSPA) paradigm--a dual encoder projection alignment mechanism enabling joint optimizing and modeling native ECG signal-image within a shared feature space. HeartcareGPT achieves consistent improvements across diverse ECG understanding tasks, validating both the effectiveness of the unified modeling paradigm and the necessity of a high-quality data pipeline, and establishing a methodological foundation for extending Med-MLLMs towards physiological signal domains. Our project is available at https://github.com/ZJU4HealthCare/HeartcareGPT .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。