arXiv:2411.08756cs.CV2024-11

用掩码图像建模提升半监督语义分割,通过类内重建减少语义混淆。

Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation

  • 按类别独立重建图像区域,增强类内像素关联。
  • 在COCO、PASCAL VOC上达到新最佳,性能超越现有方法。
  • 适合做半监督分割研究或想提升模型泛化能力的开发者。

鉴于半监督与自监督学习共享核心原理——有效利用无标签数据中的知识,当前多数半监督语义分割方法已融合主流自监督范式以实现进一步正则化。然而,前沿生成式自监督范式‘掩码图像建模’(Masked Image Modeling)的潜力尚未被充分挖掘。该范式通过像素重建过程建立被遮蔽区域与可见区域间的关联。本文基于此思想,提出一种新型类级掩码图像建模方法,根据图像区域所属类别独立进行重建,使掩码诱导的关联局限于同一类别内部,从而缓解基础方法中因整体重建引发的语义混淆问题。为进一步强化类内关联,设计了一种特征聚合策略,最小化同一类别下遮蔽与可见部分特征间的距离。此外,在语义空间中探索了掩码图像建模的应用以增强正则化效果。大量实验在COCO、PASCAL VOC等知名基准上验证了本方法的优越性,性能达到当前最优水平。代码将公开于 https://github.com/haoxt/S4MIM。

原文摘要 · Abstract (English)

In view of the fact that semi- and self-supervised learning share a fundamental principle, effectively modeling knowledge from unlabeled data, various semi-supervised semantic segmentation methods have integrated representative self-supervised learning paradigms for further regularization. However, the potential of the state-of-the-art generative self-supervised paradigm, masked image modeling, has been scarcely studied. This paradigm learns the knowledge through establishing connections between the masked and visible parts of masked image, during the pixel reconstruction process. By inheriting and extending this insight, we successfully leverage masked image modeling to boost semi-supervised semantic segmentation. Specifically, we introduce a novel class-wise masked image modeling that independently reconstructs different image regions according to their respective classes. In this way, the mask-induced connections are established within each class, mitigating the semantic confusion that arises from plainly reconstructing images in basic masked image modeling. To strengthen these intra-class connections, we further develop a feature aggregation strategy that minimizes the distances between features corresponding to the masked and visible parts within the same class. Additionally, in semantic space, we explore the application of masked image modeling to enhance regularization. Extensive experiments conducted on well-known benchmarks demonstrate that our approach achieves state-of-the-art performance. The code will be available at https://github.com/haoxt/S4MIM.

半监督分割掩码建模自监督学习语义分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。