用分层方法增强CLIP的空间表征，提升开放词汇语义分割精度
CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation
- 分层融合早期特征与注意力图，提升分割空间连贯性
- 引入扩散模型注意力图补偿细节，使VOC mIoU达69.8%
- 无需额外训练，适用于多种开放词汇分割任务
对比语言-图像预训练（CLIP）在图像级任务中表现出强大的零样本分类能力，促使研究将其适配于无需额外训练的像素级开放词汇语义分割。核心在于改进图像级CLIP的空间表征，如用自注意力图或视觉基础模型注意力图替换最后一层自注意力图。本文提出一种新型分层框架CLIPer，包含早期融合模块与细粒度补偿模块。观察发现，早期层的嵌入和注意力图可保留空间结构信息，因此设计早期融合模块生成更连贯的分割图。随后，利用扩散模型的自注意力图进行局部细节补偿。在七个分割数据集上实验表明，所提方法达到当前最优性能：使用ViT-L时，VOC和COCO Object上的mIoU分别为69.8%和43.3%，分别优于ProxyCLIP 9.2%和4.1%。
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-training (CLIP) exhibits strong zero-shot classification ability on various image-level tasks, leading to the research to adapt CLIP for pixel-level open-vocabulary semantic segmentation without additional training. The key is to improve spatial representation of image-level CLIP, such as replacing self-attention map at last layer with self-self attention map or vision foundation model based attention map. In this paper, we present a novel hierarchical framework, named CLIPer, that hierarchically improves spatial representation of CLIP. The proposed CLIPer includes an early-layer fusion module and a fine-grained compensation module. We observe that, the embeddings and attention maps at early layers can preserve spatial structural information. Inspired by this, we design the early-layer fusion module to generate segmentation map with better spatial coherence. Afterwards, we employ a fine-grained compensation module to compensate the local details using the self-attention maps of diffusion model. We conduct the experiments on seven segmentation datasets. Our proposed CLIPer achieves the state-of-the-art performance on these datasets. For instance, using ViT-L, CLIPer has the mIoU of 69.8% and 43.3% on VOC and COCO Object, outperforming ProxyCLIP by 9.2% and 4.1% respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。