CLIP在多物体识别中存在严重偏见，大小和顺序影响判断结果。
CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation
- 通过新数据集ComCO分析CLIP对多物体的表征机制
- 发现图像编码器偏好大物体，文本编码器倾向首个提及对象
- 提示顺序改变会导致生成图像中物体主次颠倒，适合关注模型偏差的研究者
对比语言-图像预训练（CLIP）模型在零样本分类中表现优异，但在复杂多物体场景下面临挑战。本研究利用专为评估多物体场景设计的新数据集ComCO，全面分析了CLIP在该类任务中的局限性。结果揭示显著偏差：文本编码器优先关注首提物体，图像编码器更青睐较大物体。通过检索与分类任务，我们在多个CLIP变体上量化了这些偏差，并追溯其根源至CLIP训练过程，结合对LAION数据集及训练进程的分析。图像-文本匹配实验显示，当物体大小或词序变化时，性能明显下降，凸显CLIP在语义相似但重述提示下的不稳定性。进一步扩展至长提示和文生图模型如Stable Diffusion，证实提示顺序会影响生成图像中物体的突出程度。项目代码与数据集详见：https://clip-oscope.github.io。
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-training (CLIP) models excel in zero-shot classification, yet face challenges in complex multi-object scenarios. This study offers a comprehensive analysis of CLIP's limitations in these contexts using a specialized dataset, ComCO, designed to evaluate CLIP's encoders in diverse multi-object scenarios. Our findings reveal significant biases: the text encoder prioritizes first-mentioned objects, and the image encoder favors larger objects. Through retrieval and classification tasks, we quantify these biases across multiple CLIP variants and trace their origins to CLIP's training process, supported by analyses of the LAION dataset and training progression. Our image-text matching experiments show substantial performance drops when object size or token order changes, underscoring CLIP's instability with rephrased but semantically similar captions. Extending this to longer captions and text-to-image models like Stable Diffusion, we demonstrate how prompt order influences object prominence in generated images. For more details and access to our dataset and analysis code, visit our project repository: https://clip-oscope.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。