arXiv:2511.03196cs.LGstat.ML2025-11ICML被引 1

用概率耦合模型更好对齐多模态数据,提升医疗信息融合效果。

Cross-Modal Alignment via Variational Copula Modelling

论文配图:Cross-Modal Alignment via Variational Copula Modelling
图 1 · 摘自论文原文
  • 基于高斯混合与耦合模型,建模多模态间复杂交互关系。
  • 在MIMIC数据集上,对缺失模态的重建准确率显著优于基线方法。
  • 适合医疗健康等需融合异构数据的场景,尤其关注模态缺失问题。

真实世界应用中常见多种数据模态(如电子病历、医学影像和临床文本)。为整合多源信息,需发展多模态学习方法以对齐并融合不同模态的表示。现有方法主要依赖拼接或克罗内克积,过度简化了模态间的交互结构,且对高阶交互下的潜在表示联合分布研究不足。耦合模型可有效刻画变量间交互,天然连接联合分布与边缘分布。本文提出一种新型耦合驱动的多模态学习框架,聚焦于学习各模态的联合分布以捕捉复杂交互。核心思想是将耦合模型视为高效对齐各模态边缘分布的工具。通过假设每模态服从高斯混合分布,并在联合分布上引入耦合模型,本方法可生成缺失模态的精准表示。在公开MIMIC数据集上的大量实验表明,该模型性能显著优于其他对比方法。代码已开源:https://github.com/HKU-MedAI/CMCM。

原文摘要 · Abstract (English)

Various data modalities are common in real-world applications (e.g., electronic health records, medical images and clinical notes in healthcare). It is essential to develop multimodal learning methods to aggregate various information from multiple modalities. The main challenge is how to appropriately align and fuse the representations of different modalities into a joint distribution. Existing methods mainly rely on concatenation or the Kronecker product, oversimplifying the interaction structure between modalities and indicating a need to model more complex interactions. Additionally, the joint distribution of latent representations with higher-order interactions is underexplored. Copula is a powerful statistical structure for modelling the interactions among variables, as it naturally bridges the joint distribution and marginal distributions of multiple variables. We propose a novel copula-driven multimodal learning framework, which focuses on learning the joint distribution of various modalities to capture the complex interactions among them. The key idea is to interpret the copula model as a tool to align the marginal distributions of the modalities efficiently. By assuming a Gaussian mixture distribution for each modality and a copula model on the joint distribution, our model can generate accurate representations for missing modalities. Extensive experiments on public MIMIC datasets demonstrate the superior performance of our model over other competitors. The code is available at https://github.com/HKU-MedAI/CMCM.

多模态学习耦合模型医疗数据分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。