提出DEARLi方法，用两个基础模型分别提升半监督全景分割的识别与定位能力。
DEARLi: Decoupled Enhancement of Recognition and Localization for Semi-supervised Panoptic Segmentation
- 用CLIP零样本分类增强识别，结合掩码变换器一致性
- 用SAM伪标签进行无类别解码器预热，提升定位精度
- 仅用158张标注图就达29.9 PQ，GPU内存需求仅为同类方法1/8
像素级标注成本高。半监督分割通过少量标注图像与大量未标注图像联合训练缓解此问题。尽管基础模型可缓解标注稀缺，其有效利用机制仍不明确。本文提出一种新型半监督全景分割方法DEARLi，基于两个专用基础模型：通过零样本分类补充无监督掩码变换器一致性以增强识别；通过类无关解码器预热（基于SAM伪标签）提升定位。该解耦增强策略在标签极少、类别繁多的挑战场景中表现突出。在仅使用158张标注图像的情况下，ADE20K数据集上达到29.9 PQ和38.9 mIoU，显著超越当前最优方法，且训练时仅需1/8的GPU内存，仅针对全景目标训练。代码已开源。
原文摘要 · Abstract (English)
Pixel-level annotation is expensive and time-consuming. Semi-supervised segmentation methods address this challenge by learning models on few labeled images alongside a large corpus of unlabeled images. Although foundation models could further account for label scarcity, effective mechanisms for their exploitation remain underexplored. We address this by devising a novel semi-supervised panoptic approach fueled by two dedicated foundation models. We enhance recognition by complementing unsupervised mask-transformer consistency with zero-shot classification of CLIP features. We enhance localization by class-agnostic decoder warm-up with respect to SAM pseudo-labels. The resulting decoupled enhancement of recognition and localization (DEARLi) particularly excels in the most challenging semi-supervised scenarios with large taxonomies and limited labeled data. Moreover, DEARLi outperforms the state of the art in semi-supervised semantic segmentation by a large margin while requiring 8x less GPU memory, in spite of being trained only for the panoptic objective. We observe 29.9 PQ and 38.9 mIoU on ADE20K with only 158 labeled images. The source code is available at https://github.com/helen1c/DEARLi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。