用对比学习精准生成缺失脑肿瘤MRI,提升分割效果
Multi-modal Contrastive Learning for Tumor-specific Missing Modality Synthesis
- 基于熵选择特征的多模态对比学习,聚焦肿瘤区域
- 生成缺失影像同时预测分割图,提升肿瘤细节保真度
- 适合医学影像合成与肿瘤分割任务的研究者
多模态磁共振成像(MRI)对脑部解剖与病灶信息互补至关重要,但临床中因时间、成本和患者移动等因素难以获取高质量多模态数据。为解决此问题,我们提出一种生成模型,通过融合多模态对比学习并聚焦关键肿瘤区域来合成缺失模态图像。具体而言,设计针对多种源模态的对比学习机制,并在过程中基于特征熵进行选择以增强有效性;网络不仅生成缺失目标模态图像,还同步输出分割结果。结合对比损失、分割损失与自表示损失,有效捕捉目标特异性信息,生成高质量图像。在脑部MR图像合成挑战赛中,该模型在缺失模态生成任务上表现优异。
原文摘要 · Abstract (English)
Multi-modal magnetic resonance imaging (MRI) is essential for providing complementary information about brain anatomy and pathology, leading to more accurate diagnoses. However, obtaining high-quality multi-modal MRI in a clinical setting is difficult due to factors such as time constraints, high costs, and patient movement artifacts. To overcome this difficulty, there is increasing interest in developing generative models that can synthesize missing target modality images from the available source ones. Therefore, our team, PLAVE, design a generative model for missing MRI that integrates multi-modal contrastive learning with a focus on critical tumor regions. Specifically, we integrate multi-modal contrastive learning, tailored for multiple source modalities, and enhance its effectiveness by selecting features based on entropy during the contrastive learning process. Additionally, our network not only generates the missing target modality images but also predicts segmentation outputs, simultaneously. This approach improves the generator's capability to precisely generate tumor regions, ultimately improving performance in downstream segmentation tasks. By leveraging a combination of contrastive, segmentation, and additional self-representation losses, our model effectively reflects target-specific information and generate high-quality target images. Consequently, our results in the Brain MR Image Synthesis challenge demonstrate that the proposed model excelled in generating the missing modality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。