解决医学影像多模态缺失时的不平衡问题,提升分割精度。
DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation
- 用动态注意力与自适应掩码抑制缺失模态干扰
- 在两个数据集上均超越现有方法,显著提升分割效果
- 适合处理临床中常见模态不完整场景的医生和研究者
不完整的多模态医学图像分割面临模态不平衡的挑战,包括不同的模态缺失率和异质的模态贡献。由于依赖于理想化的完全模态可用假设,现有方法无法动态平衡各模态贡献,且忽略模态间的结构关系,导致在真实临床场景中表现不佳。为此,我们提出一种新模型——动态模态感知融合网络(DMAF-Net)。该模型包含三个核心设计:首先,引入动态模态感知融合(DMAF)模块,通过结合Transformer注意力与自适应掩码,利用注意力图动态调整各模态贡献,抑制缺失模态的干扰;其次,设计协同关系蒸馏与原型蒸馏框架,通过协方差一致性与掩码图注意力实现全局-局部特征对齐,并通过跨模态类别特定原型对齐保证语义一致性;第三,提出动态训练监控(DTM)策略,实时追踪蒸馏差距,自适应重加权损失并缩放梯度,以稳定优化过程并平衡不同模态的收敛速度。在BraTS2020和MyoPS2020数据集上的大量实验表明,DMAF-Net在不完整多模态医学图像分割任务中优于现有方法。代码已开源。
原文摘要 · Abstract (English)
Incomplete multi-modal medical image segmentation faces critical challenges from modality imbalance, including imbalanced modality missing rates and heterogeneous modality contributions. Due to their reliance on idealized assumptions of complete modality availability, existing methods fail to dynamically balance contributions and neglect the structural relationships between modalities, resulting in suboptimal performance in real-world clinical scenarios. To address these limitations, we propose a novel model, named Dynamic Modality-Aware Fusion Network (DMAF-Net). The DMAF-Net adopts three key ideas. First, it introduces a Dynamic Modality-Aware Fusion (DMAF) module to suppress missing-modality interference by combining transformer attention with adaptive masking and weight modality contributions dynamically through attention maps. Second, it designs a synergistic Relation Distillation and Prototype Distillation framework to enforce global-local feature alignment via covariance consistency and masked graph attention, while ensuring semantic consistency through cross-modal class-specific prototype alignment. Third, it presents a Dynamic Training Monitoring (DTM) strategy to stabilize optimization under imbalanced missing rates by tracking distillation gaps in real-time, and to balance convergence speeds across modalities by adaptively reweighting losses and scaling gradients. Extensive experiments on BraTS2020 and MyoPS2020 demonstrate that DMAF-Net outperforms existing methods for incomplete multi-modal medical image segmentation. Extensive experiments on BraTS2020 and MyoPS2020 demonstrate that DMAF-Net outperforms existing methods for incomplete multi-modal medical image segmentation. Our code is available at https://github.com/violet-42/DMAF-Net.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。