ProbMED用概率建模实现多模态医学数据精准对齐
ProbMed: A Probabilistic Framework for Medical Multimodal Binding
- 采用概率对比学习，将四种医学模态映射到统一概率嵌入空间
- 在13个数据集上超越现有模型，在零样本和少样本分类中表现优异
- 适合需要融合影像与文本等多源医疗数据的临床决策研究
医学决策需整合多种信息，包括影像与临床文本。这些模态常以多对多方式采集，但现有医学视觉-语言预训练模型（Med-VLPMs）未在训练中考虑这种映射关系。为此，我们提出概率模态增强诊断模型（ProbMED），采用概率对比学习，建模嵌入分布而非确定性估计。该模型将胸部X光片、心电图、超声心动图和临床文本四类模态对齐至统一概率嵌入空间，使用带希尔伯特距离的InfoNCE损失融合跨模态分布，并引入概率合成采样损失，捕捉各模态的均值与方差，增强模态内绑定。在13个医学数据集上的大量实验表明，本模型在跨模态检索、零样本与少样本分类任务中优于现有Med-VLPMs。同时，验证了多模态融合在预后预测中的鲁棒性，显著提升模态内与跨模态绑定效果。
原文摘要 · Abstract (English)
Medical decision-making requires integrating diverse medical information, from imaging to clinical narratives. These medical modalities are often acquired in a many-to-many manner. However, current medical vision-language pretraining models (Med-VLPMs) fail to directly account for this many-to-many mapping in their model training and embeddings. To address this, we present Probabilistic Modality-Enhanced Diagnosis (ProbMED), a multimodal Med-VLPM that employs probabilistic contrastive learning to model distributions over embeddings rather than deterministic estimates. ProbMED aligns four distinct modalities -- chest X-rays, electrocardiograms, echocardiograms, and clinical text -- into a unified probabilistic embedding space. We use InfoNCE loss with Hellinger distance to integrate inter-modality distributions. We introduce a probabilistic synthetic sampling loss that captures modality-specific mean and variance to improve intra-modality binding. Extensive experiments across 13 medical datasets demonstrate that our model outperforms current Med-VLPMs in cross-modality retrieval, zero-shot, and few-shot classification. We also demonstrate the robust integration of multiple modalities for prognostication, showing improved intra- and inter-medical modality binding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。