arXiv:2411.14795cs.CL2024-11

让心电图大模型真正理解信号,而非依赖病历文字误导判断。

De-biased Multimodal Electrocardiogram Analysis

  • 直接输入心电图嵌入向量,保留更多原始信息并激活大模型推理能力
  • 在对抗测试中表现优异,零样本下仍能准确分析心电图差异
  • 通过因果去偏设计,消除病情严重度带来的虚假关联,适合临床真实场景

多模态大语言模型(MLLM)在医学影像中应用日益广泛,但在关键的临床心电图(ECG)分析中仍面临挑战。现有方法将心电图转为文本标签,大幅压缩信息且未能发挥大模型推理优势。本文通过投影层直接将心电图嵌入送入大模型,保留更多信号特征,并支持跨时间点心电图对比。研究发现,现有模型常因病情严重度这一混淆因子,产生问题与答案间的虚假关联,从而忽略心电图输入。为此,我们基于反向调整理论设计去偏预训练方法,消除混淆因子影响。实验表明,该模型在对抗性测试中表现稳健,具备零样本能力;随机心电图测试也验证了其对真实信号的有效理解。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) are increasingly being applied in the medical field, particularly in medical imaging. However, developing MLLMs for ECG signals, which are crucial in clinical settings, has been a significant challenge beyond medical imaging. Previous studies have attempted to address this by converting ECGs into several text tags using an external classifier in a training-free manner. However, this approach significantly compresses the information in ECGs and underutilizes the reasoning capabilities of LLMs. In this work, we directly feed the embeddings of ECGs into the LLM through a projection layer, retaining more information about ECGs and better leveraging the reasoning abilities of LLMs. Our method can also effectively handle a common situation in clinical practice where it is necessary to compare two ECGs taken at different times. Recent studies found that MLLMs may rely solely on text input to provide answers, ignoring inputs from other modalities. We analyzed this phenomenon from a causal perspective in the context of ECG MLLMs and discovered that the confounder, severity of illness, introduces a spurious correlation between the question and answer, leading the model to rely on this spurious correlation and ignore the ECG input. Such models do not comprehend the ECG input and perform poorly in adversarial tests where different expressions of the same question are used in the training and testing sets. We designed a de-biased pre-training method to eliminate the confounder's effect according to the theory of backdoor adjustment. Our model performed well on the ECG-QA task under adversarial testing and demonstrated zero-shot capabilities. An interesting random ECG test further validated that our model effectively understands and utilizes the input ECG signal.

心电图分析多模态去偏学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。