arXiv:2410.17565cs.CV2024-10

提出双银行一致性框架,实现多模态医学图像分割的高效学习。

Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation

  • 设计可扩展的全模态统一网络,支持任意数量模态输入。
  • 引入可学习的调制与原型库,同时捕捉共性与特异性特征。
  • 无需生成模型,通过双层次一致性增强未标注数据利用效率。

多模态半监督学习在医学图像分割中备受关注,因其能有效利用多源影像数据并减少对标注的依赖。现有方法面临三大挑战:(1) 网络结构复杂,难以扩展至两模态以上;(2) 忽视模态特异性特征,仅关注不变表示;(3) 依赖生成模型处理未标注数据,可靠性不足。为此,本文提出双银行双一致性(DBDC)框架。首先设计模态全包容分割网络,支持任意数量模态输入。其次构建可学习的模态级调制库(MLMB)和原型库(MLPB),结合模态原型对比学习(MPCL)联合优化。进一步引入模态自适应加权(MAW),动态调整各模态学习权重以平衡训练进度。最后设计双一致性(DC)策略,在图像与特征层面强制一致性,不依赖生成模型。在包含三组开源数据集的2至4模态分割任务上评估,实验表明该方法优于当前最优基准。

原文摘要 · Abstract (English)

Multi-modality (MM) semi-supervised learning (SSL) based medical image segmentation has recently gained increasing attention for its ability to utilize MM data and reduce reliance on labeled images. However, current methods face several challenges: (1) Complex network designs hinder scalability to scenarios with more than two modalities. (2) Focusing solely on modality-invariant representation while neglecting modality-specific features, leads to incomplete MM learning. (3) Leveraging unlabeled data with generative methods can be unreliable for SSL. To address these problems, we propose Double Bank Dual Consistency (DBDC), a novel MM-SSL approach for medical image segmentation. To address challenge (1), we propose a modality all-in-one segmentation network that accommodates data from any number of modalities, removing the limitation on modality count. To address challenge (2), we design two learnable plug-in banks, Modality-Level Modulation bank (MLMB) and Modality-Level Prototype (MLPB) bank, to capture both modality-invariant and modality-specific knowledge. These banks are updated using our proposed Modality Prototype Contrastive Learning (MPCL). Additionally, we design Modality Adaptive Weighting (MAW) to dynamically adjust learning weights for each modality, ensuring balanced MM learning as different modalities learn at different rates. Finally, to address challenge (3), we introduce a Dual Consistency (DC) strategy that enforces consistency at both the image and feature levels without relying on generative methods. We evaluate our method on a 2-to-4 modality segmentation task using three open-source datasets, and extensive experiments show that our method outperforms state-of-the-art approaches.

多模态半监督医学图像一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。