用无标签图像和分割掩码让CLIP学会像素级语义理解
Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
- 用SAM/DINO生成的掩码引导CLIP学习物体位置
- 在线聚类自动生成通用语义概念，无需人工标注
- 在开放词汇分割上超越CLIP，媲美有标注监督方法
大型视觉-语言模型如CLIP在图像级任务中展现出出色的开放词汇能力，能够识别图像中存在哪些物体。然而，在需要定位物体位置的像素级任务（如语义分割）中表现不佳。本文提出PixelCLIP，通过使用无标签图像和由SAM、DINO等视觉基础模型生成的掩码，引导CLIP图像编码器学习像素级理解。为解决缺乏语义标签时利用掩码的挑战，我们设计了一种基于可学习类别名的在线聚类算法，以获取通用语义概念。PixelCLIP在开放词汇语义分割任务上显著优于CLIP，性能与基于标题监督的方法相当。
原文摘要 · Abstract (English)
Large-scale vision-language models like CLIP have demonstrated impressive open-vocabulary capabilities for image-level tasks, excelling in recognizing what objects are present. However, they struggle with pixel-level recognition tasks like semantic segmentation, which additionally require understanding where the objects are located. In this work, we propose a novel method, PixelCLIP, to adapt the CLIP image encoder for pixel-level understanding by guiding the model on where, which is achieved using unlabeled images and masks generated from vision foundation models such as SAM and DINO. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm using learnable class names to acquire general semantic concepts. PixelCLIP shows significant performance improvements over CLIP and competitive results compared to caption-supervised methods in open-vocabulary semantic segmentation. Project page is available at https://cvlab-kaist.github.io/PixelCLIP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。