arXiv:2411.19331cs.CVcs.AI2024-11ICCV被引 48

用语言桥接DINO与CLIP,实现更精准的开放词汇分割。

Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation

  • 通过映射函数对齐CLIP文本嵌入与DINOv2图像块特征。
  • 在无监督基准上达到当前最优性能,分割更自然、噪声更少。
  • 无需微调主干模型,适合需要高精度分割的场景。

开放词汇分割(OVS)旨在对自由文本概念进行图像分割,而无需预定义类别。现有视觉语言模型如CLIP虽能利用视觉变压器的粗略空间信息生成分割掩码,但因图像与文本特征全局对齐,存在空间定位不足的问题。反观自监督视觉模型如DINO在细粒度视觉编码方面表现优异,却缺乏与语言的融合能力。为此,本文提出Talk2DINO,一种结合DINOv2空间精度与CLIP语言理解能力的混合方法。通过一个学习的映射函数,将CLIP的文本嵌入对齐至DINOv2的图像块级特征,且无需微调主干网络。训练时,利用DINOv2的注意力图选择性地对齐局部视觉块与文本嵌入。实验表明,该方法显著提升分割质量,生成更自然、噪声更少的分割结果,并能有效区分前景与背景。在多个无监督OVS基准上,Talk2DINO均达到当前最优水平。代码与模型已公开于:https://lorebianchi98.github.io/Talk2DINO/。

原文摘要 · Abstract (English)

Open-Vocabulary Segmentation (OVS) aims at segmenting images from free-form textual concepts without predefined training classes. While existing vision-language models such as CLIP can generate segmentation masks by leveraging coarse spatial information from Vision Transformers, they face challenges in spatial localization due to their global alignment of image and text features. Conversely, self-supervised visual models like DINO excel in fine-grained visual encoding but lack integration with language. To bridge this gap, we present Talk2DINO, a novel hybrid approach that combines the spatial accuracy of DINOv2 with the language understanding of CLIP. Our approach aligns the textual embeddings of CLIP to the patch-level features of DINOv2 through a learned mapping function without the need to fine-tune the underlying backbones. At training time, we exploit the attention maps of DINOv2 to selectively align local visual patches with textual embeddings. We show that the powerful semantic and localization abilities of Talk2DINO can enhance the segmentation process, resulting in more natural and less noisy segmentations, and that our approach can also effectively distinguish foreground objects from the background. Experimental results demonstrate that Talk2DINO achieves state-of-the-art performance across several unsupervised OVS benchmarks. Source code and models are publicly available at: https://lorebianchi98.github.io/Talk2DINO/.

开放词汇分割视觉语言对齐自监督学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。