通过互知识分析，揭示CLIP模型零样本分类的内在机制
Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge
- 基于文本概念解释，挖掘视觉与语言模态的共同学习特征
- 覆盖13种不同架构和预训练数据的CLIP模型，验证方法普适性
- 提供可理解的决策解释，适合想搞懂CLIP行为的研究者
对比语言-图像预训练（CLIP）通过将图像与文本类别表示映射到共享嵌入空间，实现零样本图像分类，即检索与图像最接近的类别。本文从双模态互知识视角，提出新方法解析CLIP的分类逻辑：视觉与语言编码器共同学习了哪些概念，影响了联合嵌入空间中样本的相对距离？我们采用基于文本概念的解释方法，验证其有效性，并对13种不同架构、规模及预训练数据集的CLIP模型进行系统分析。研究探索了多种因素与互知识的关系，深入剖析零样本预测过程。结果表明，该方法能有效且直观地理解CLIP的零样本分类决策。
原文摘要 · Abstract (English)
Contrastive Language-Image Pretraining (CLIP) performs zero-shot image classification by mapping images and textual class representation into a shared embedding space, then retrieving the class closest to the image. This work provides a new approach for interpreting CLIP models for image classification from the lens of mutual knowledge between the two modalities. Specifically, we ask: what concepts do both vision and language CLIP encoders learn in common that influence the joint embedding space, causing points to be closer or further apart? We answer this question via an approach of textual concept-based explanations, showing their effectiveness, and perform an analysis encompassing a pool of 13 CLIP models varying in architecture, size and pretraining datasets. We explore those different aspects in relation to mutual knowledge, and analyze zero-shot predictions. Our approach demonstrates an effective and human-friendly way of understanding zero-shot classification decisions with CLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。