CLIP的语义空间有根本缺陷，新方法用相似度图修复了其表达能力
Is CLIP ideal? No. Can we fix it? Yes!
- 提出密集余弦相似度图，保留图像与文本的拓扑结构
- 在多个基准上超越经典CLIP模型，提升语义理解能力
- 适合研究多模态表示或改进视觉语言模型的学者
对比语言-图像预训练（CLIP）是一种广泛使用的多模态表征学习方法，具有良好的语义组织性。然而，其潜在空间在处理复杂视觉-文本交互时表现不佳。现有工作尝试通过数据或算法改进，但问题可能源于几何结构本身。本文严格分析了CLIP的潜在空间特性，证明不存在任何类似CLIP的联合嵌入空间能同时正确表达四项内容：1. 基本描述与图像内容；2. 属性绑定；3. 空间位置与关系；4. 否定。基于此，我们提出密集余弦相似度图（DCSMs），一种可解释的评分机制，保留图像块与文本标记间的语义拓扑。该方法显著提升经典CLIP类模型在多个基准上的性能。代码与数据已开源：https://github.com/Raphoo/DCSM_Ideal_CLIP
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-Training (CLIP) is a popular method for learning multimodal latent spaces with well-organized semantics. Despite its wide range of applications, CLIP's latent space is known to fail at handling complex visual-textual interactions. Recent works attempt to address its shortcomings with data-centric or algorithmic approaches. But what if the problem is more fundamental, and lies in the geometry of CLIP? Toward this end, we rigorously analyze CLIP's latent space properties, and prove that no CLIP-like joint embedding space exists which can correctly do any two of the following at the same time: 1. represent basic descriptions and image content, 2. represent attribute binding, 3. represent spatial location and relationships, 4. represent negation. Informed by this analysis, we propose Dense Cosine Similarity Maps (DCSMs) as a principled and interpretable scoring method for CLIP-like models, which solves the fundamental limitations of CLIP by retaining the semantic topology of the image patches and text tokens. This method improves upon the performance of classical CLIP-like joint encoder models on a wide array of benchmarks. We share our code and data here for reproducibility: https://github.com/Raphoo/DCSM_Ideal_CLIP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。