对比发现，语言监督让CLIP在视觉-语言任务中表现更优。
Data or Language Supervision: What Makes CLIP Better than DINO?
- 控制变量下对比CLIP与DINO的预训练效果
- CLIP在文本相关任务上显著优于DINO，而DINO略胜于视觉主导任务
- 语言监督比数据量对模型性能影响更大
CLIP作为视觉-语言模型的视觉编码器，表现优于自监督模型DINO，但其优势源于语言监督还是更大规模数据尚不明确。为厘清这一问题，我们在相同架构、数据集和训练配置下对CLIP与DINO进行预训练，使其在ImageNet上达到相近准确率。嵌入分析显示，CLIP能捕捉高级语义（如物体类别、文本），而DINO更关注颜色、风格等低层特征。当集成至20个VQA基准测试时，CLIP在文本密集型任务中表现更佳，而DINO在视觉主导任务中略有优势。不同语言监督形式（如Sigmoid损失、预训练语言编码器）带来的提升有限。研究揭示了视觉编码器设计对视觉-语言模型性能的影响机制。
原文摘要 · Abstract (English)
CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs), but it remains unclear whether this advantage stems from CLIP's language supervision or its much larger training data. To disentangle these factors, we pre-train CLIP and DINO under controlled settings -- using the same architecture, dataset, and training configuration -- achieving similar ImageNet accuracy. Embedding analysis shows that CLIP captures high-level semantics (e.g., object categories, text), while DINO is more responsive to low-level features like colors and styles. When integrated into VLMs and evaluated on 20 VQA benchmarks, CLIP excels at text-intensive tasks, while DINO slightly outperforms on vision-centric ones. Variants of language supervision (e.g., sigmoid loss, pre-trained language encoders) yield limited gains. Our findings provide scientific insights into vision encoder design and its impact on VLM performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。