arXiv:2509.16170cs.CV2025-09NeurIPS被引 5

统一处理多模态图像分割中缺失模态问题,无需为每种组合训练单独模型。

UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation

  • 通过分层自监督补偿,跨输入、特征、输出三阶段弥合完整与缺失模态的表征差距。
  • 在脑肿瘤分割等任务中,对所有缺失模态组合均显著优于现有方法,性能波动小。
  • 适合实际部署中模态缺失场景,如医疗影像设备故障或传感器丢失时使用。

多模态图像分割在真实场景中面临因模态不全或损坏导致性能下降的问题。现有方法虽通过每种模态组合专用模型缓解训练-推理模态差异,但需大量模型子集和匹配,部署成本高。本文提出统一模态松弛分割网络UniMRSeg,基于分层自监督补偿(HSSC)。该方法在输入、特征、输出三个层级上逐步弥合完整与缺失模态间的表示差距:首先采用混合打乱掩码增强进行模态重建,促使模型学习模态固有特性并通过跨模态融合生成缺失模态的有意义表示;其次通过模态不变对比学习隐式补偿不完整-完整模态对的特征空间距离;进一步引入轻量级反向注意力适配器,显式补偿冻结编码器中弱感知语义;最后在混合一致性约束下微调,确保所有模态组合下预测稳定,性能波动小。无需额外组件,UniMRSeg在基于MRI的脑肿瘤分割、RGB-D语义分割、RGB-D/T显著目标分割任务中均显著超越当前最优方法。代码将发布于https://github.com/Xiaoqi-Zhao-DLUT/UniMRSeg。

原文摘要 · Abstract (English)

Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance. While existing methods address training-inference modality gaps via specialized per-combination models, they introduce high deployment costs by requiring exhaustive model subsets and model-modality matching. In this work, we propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC). Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels. % First, we adopt modality reconstruction with the hybrid shuffled-masking augmentation, encouraging the model to learn the intrinsic modality characteristics and generate meaningful representations for missing modalities through cross-modal fusion. % Next, modality-invariant contrastive learning implicitly compensates the feature space distance among incomplete-complete modality pairs. Furthermore, the proposed lightweight reverse attention adapter explicitly compensates for the weak perceptual semantics in the frozen encoder. Last, UniMRSeg is fine-tuned under the hybrid consistency constraint to ensure stable prediction under all modality combinations without large performance fluctuations. Without bells and whistles, UniMRSeg significantly outperforms the state-of-the-art methods under diverse missing modality scenarios on MRI-based brain tumor segmentation, RGB-D semantic segmentation, RGB-D/T salient object segmentation. The code will be released at https://github.com/Xiaoqi-Zhao-DLUT/UniMRSeg.

多模态分割自监督学习医学影像鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。