用合成负样本提升CLIP的组合推理能力
TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
- 通过上下文学习生成难负文本，再用文生图模型合成对应负图像
- 在CC3M和CC12M数据集上使糖类识别任务准确率提升超9%
- 适合关注视觉语言模型泛化与推理能力的研究者
对比语言-图像预训练（CLIP）模型通过最大化文本与视觉模态间的互信息来学习表征。训练数据的性质直接影响其下游任务表现。然而，当前图像-文本数据集缺乏组合多样性，限制了CLIP的组合推理能力。我们提出通过上下文学习生成‘难负’文本，并利用文生图模型合成对应负图像，构建硬负样本。引入一种新的交替式对比预训练策略，联合使用这些合成负样本训练CLIP。实验表明，所提方法TripletCLIP在现有数据集（如CC3M、CC12M）上显著增强CLIP的组合能力，在同等计算预算下使SugarCrepe基准测试准确率绝对提升超过9%，同时在零样本图像分类与图像检索任务中也取得改进。代码、模型与数据已公开。
原文摘要 · Abstract (English)
Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. This makes the nature of the training data a significant factor in the efficacy of CLIP for downstream tasks. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating ``hard'' negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution. We introduce a novel contrastive pre-training strategy that leverages these hard negative captions and images in an alternating fashion to train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-shot image classification and image retrieval. Our code, models, and data are available at: https://tripletclip.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。