arXiv:2506.22032cs.CV2025-06被引 1

用部分CLIP提升零样本分割,兼顾空间精度与语义对齐。

Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation

  • 将分割模型与CLIP语义头结合,实现空间精度与语义对齐的融合。
  • 在两个基准上分别提升0.9%和1.2%的hIoU,显著优于现有方法。
  • 适合追求高效零样本分割、不依赖标注数据的研究者。

零样本语义分割(ZSS)旨在仅使用已见类的监督信息,对已见和未见类别进行分割。除基于适配的方法外,基于知识蒸馏的方法尝试将视觉-语言模型(如CLIP)的视觉-语言对齐能力迁移至分割模型。然而,这种知识迁移仍面临两大挑战:(1) 视觉特征与文本空间对齐困难,需同时具备空间精度与视觉-语言对齐;(2) CLIP的全局表示与分割模型的局部细粒度特征之间存在语义鸿沟。为解决挑战(1),本文提出Chimera-Seg,其结构类似希腊神话中的奇美拉,将分割骨干作为主体,基于CLIP的语义头作为头部,整合空间精度与语义对齐。具体地,Chimera-Seg包含可训练的分割模型与一个CLIP语义头(CSH),后者将密集特征映射到与CLIP对齐的空间。CSH融合了冻结的CLIP视觉编码器子网络和固定投影层,以及轻量级可训练组件。部分来自CLIP视觉编码器的模块与分割模型协同工作,在保留分割能力的同时,简化向CLIP语义空间的映射。为解决挑战(2),提出选择性全局蒸馏(SGD),从与CLIP CLS token高度相似的密集特征中蒸馏知识,并随训练进程逐步减少用于对齐的特征数量。此外,还引入语义对齐模块(SAM),进一步对齐密集视觉特征与冻结的CLIP文本编码器提取的语义嵌入。在两个基准测试中,平均交并比(hIoU)分别提升了0.9%和1.2%。

原文摘要 · Abstract (English)

Zero-shot Semantic Segmentation (ZSS) aims to segment both seen and unseen classes using supervision from only seen classes. Beyond adaptation-based methods, distillation-based approaches transfer vision-language alignment of vision-language model, e.g., CLIP, to segmentation models. However, such knowledge transfer remains challenging due to: (1) the difficulty of aligning vision-based features with the textual space, which requires combining spatial precision with vision-language alignment; and (2) the semantic gap between CLIP's global representations and the local, fine-grained features of segmentation models. To address challenge (1), we propose Chimera-Seg, which integrates a segmentation backbone as the body and a CLIP-based semantic head as the head, like the Chimera in Greek mythology, combining spatial precision with vision-language alignment. Specifically, Chimera-Seg comprises a trainable segmentation model and a CLIP Semantic Head (CSH), which maps dense features into the CLIP-aligned space. The CSH incorporates a frozen subnetwork and fixed projection layers from the CLIP visual encoder, along with lightweight trainable components. The partial module from CLIP visual encoder, paired with the segmentation model, retains segmentation capability while easing the mapping to CLIP's semantic space. To address challenge (2), we propose Selective Global Distillation (SGD), which distills knowledge from dense features exhibiting high similarity to the CLIP CLS token, while gradually reducing the number of features used for alignment as training progresses. Besides, we also use a Semantic Alignment Module (SAM) to further align dense visual features with semantic embeddings extracted from the frozen CLIP text encoder. Experiments on two benchmarks show improvements of 0.9% and 1.2% in hIoU.

零样本分割视觉语言对齐知识蒸馏语义分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。