arXiv:2409.17055cs.AIcs.LG2024-09被引 13

提出DRIM模型,从不完整多模态医疗数据中分离共享与特异表征。

DRIM: Learning Disentangled Representations from Incomplete Multimodal Healthcare Data

  • 通过分离共享与模态特异表征,提升多模态医疗数据利用效率。
  • 在胶质瘤生存预测任务上超越现有方法,对缺失模态鲁棒。
  • 适用于临床数据不全场景,适合医学人工智能研究者使用。

真实医疗数据通常是多模态且不完整的,亟需高效整合多种数据的深度学习模型。不同模态如病理切片、MRI和基因数据,为改善预后预测和发现新治疗路径提供了前所未有的机会。对比学习广泛用于多模态任务,但其假设不同视图包含相同任务相关信息,仅利用共享信息,这在医疗数据中受限,因各模态也包含对下游任务重要的特异性知识。本文提出DRIM,一种新多模态方法,可在数据稀疏情况下捕捉共享与独特表征。具体而言,给定一组模态,目标是为每个模态编码一个可分解为两部分的表示:一部分包含跨模态的患者共性信息,另一部分包含模态特异性细节。通过增强不同患者模态间的共享信息,并最小化每模态内共享与独特成分的重叠来实现。实验显示,该方法在胶质瘤患者生存预测任务中优于现有最优算法,且对缺失模态具有鲁棒性。代码已开源,地址:https://github.com/Lucas-rbnt/DRIM。

原文摘要 · Abstract (English)

Real-life medical data is often multimodal and incomplete, fueling the growing need for advanced deep learning models capable of integrating them efficiently. The use of diverse modalities, including histopathology slides, MRI, and genetic data, offers unprecedented opportunities to improve prognosis prediction and to unveil new treatment pathways. Contrastive learning, widely used for deriving representations from paired data in multimodal tasks, assumes that different views contain the same task-relevant information and leverages only shared information. This assumption becomes restrictive when handling medical data since each modality also harbors specific knowledge relevant to downstream tasks. We introduce DRIM, a new multimodal method for capturing these shared and unique representations, despite data sparsity. More specifically, given a set of modalities, we aim to encode a representation for each one that can be divided into two components: one encapsulating patient-related information common across modalities and the other, encapsulating modality-specific details. This is achieved by increasing the shared information among different patient modalities while minimizing the overlap between shared and unique components within each modality. Our method outperforms state-of-the-art algorithms on glioma patients survival prediction tasks, while being robust to missing modalities. To promote reproducibility, the code is made publicly available at https://github.com/Lucas-rbnt/DRIM

多模态医疗AI表征学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。