让DINOv2模型实现图像与像素级图文对齐，低成本达到顶尖零样本性能。
DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment
- 用[CLS]与图块均值拼接训练文本编码器，提升图文对齐效果。
- 在零样本分类和开集语义分割上达到新SOTA，性能超越CLIP但计算成本更低。
- 适合需要高效图文对齐的开放词汇任务研究者与开发者使用。
自监督视觉基础模型生成的强大嵌入在众多下游任务中表现优异。然而，与CLIP等视觉语言模型不同，自监督视觉特征未直接与语言对齐，限制了其在开放词汇任务中的应用。本文提出的dino.txt方法为广泛使用的自监督视觉编码器DINOv2赋予了这一能力。基于LiT训练策略，我们改进了多个关键组件：将[CLS]标记与图块平均值拼接用于对齐训练，并融合文本与图像模态的数据筛选。由此构建的类CLIP模型仅需极少计算开销，即可在零样本分类与开集语义分割任务上达到当前最优性能。
原文摘要 · Abstract (English)
Self-supervised visual foundation models produce powerful embeddings that achieve remarkable performance on a wide range of downstream tasks. However, unlike vision-language models such as CLIP, self-supervised visual features are not readily aligned with language, hindering their adoption in open-vocabulary tasks. Our method, named dino.txt, unlocks this new ability for DINOv2, a widely used self-supervised visual encoder. We build upon the LiT training strategy, which trains a text encoder to align with a frozen vision model but leads to unsatisfactory results on dense tasks. We propose several key ingredients to improve performance on both global and dense tasks, such as concatenating the [CLS] token with the patch average to train the alignment and curating data using both text and image modalities. With these, we successfully train a CLIP-like model with only a fraction of the computational cost compared to CLIP while achieving state-of-the-art results in zero-shot classification and open-vocabulary semantic segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。