解决脑肿瘤分割中任意模态缺失问题,提升模型鲁棒性。
Semantic-guided Masked Mutual Learning for Multi-modal Brain Tumor Segmentation with Arbitrary Missing Modalities
- 设计双分支掩码互学习框架,通过层级一致性约束增强多模态信息互补。
- 在三个数据集上优于现有方法,在多种缺失场景下分割准确率显著提升。
- 结合SAM语义先验,有效补充缺失模态的判别性特征,适合临床实际应用。
恶性脑肿瘤是全球致死率高的严重疾病,多模态MRI对精准分割至关重要,但临床中常见模态缺失,严重影响分割性能。针对任意模态缺失下的特征学习难题,本文提出语义引导的掩码互学习(SMML)方法,通过双分支结构与层级一致性约束(HCC)实现像素级与特征级的一致性对齐,强化不完整多模态场景下的知识蒸馏。像素级约束选择并交换可靠知识,特征级约束挖掘潜在空间中的样本间与类别间关系。此外,每个学生分支集成精炼网络,利用Segment Anything Model(SAM)的语义先验提供补充信息,进一步捕获辅助判别特征。在三个挑战性脑肿瘤分割数据集上的大量实验表明,该方法在多种缺失模式下均显著优于当前最优方法。
原文摘要 · Abstract (English)
Malignant brain tumors have become an aggressive and dangerous disease that leads to death worldwide.Multi-modal MRI data is crucial for accurate brain tumor segmentation, but missing modalities common in clinical practice can severely degrade the segmentation performance. While incomplete multi-modal learning methods attempt to address this, learning robust and discriminative features from arbitrary missing modalities remains challenging. To address this challenge, we propose a novel Semantic-guided Masked Mutual Learning (SMML) approach to distill robust and discriminative knowledge across diverse missing modality scenarios.Specifically, we propose a novel dual-branch masked mutual learning scheme guided by Hierarchical Consistency Constraints (HCC) to ensure multi-level consistency, thereby enhancing mutual learning in incomplete multi-modal scenarios. The HCC framework comprises a pixel-level constraint that selects and exchanges reliable knowledge to guide the mutual learning process. Additionally, it includes a feature-level constraint that uncovers robust inter-sample and inter-class relational knowledge within the latent feature space. To further enhance multi-modal learning from missing modality data, we integrate a refinement network into each student branch. This network leverages semantic priors from the Segment Anything Model (SAM) to provide supplementary information, effectively complementing the masked mutual learning strategy in capturing auxiliary discriminative knowledge. Extensive experiments on three challenging brain tumor segmentation datasets demonstrate that our method significantly improves performance over state-of-the-art methods in diverse missing modality settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。