CLIP模型常因半真半假描述误判图像相似性，新方法有效解决此问题。
Half-Truths Break Similarity-Based Retrieval
- 将文本拆解为实体与关系单元，用反例训练模型区分正确与错误部分
- 在COCO数据集上，半真半假场景下准确率从40.6%提升至69.3%
- 适合关注图像-文本匹配鲁棒性与组合理解的研究者
当文本描述添加一个看似合理但错误的细节时，图像-文本相似度应下降。我们发现，类似CLIP的双编码器模型常违背这一直觉：添加一个看似合理但错误的物体或关系，反而可能提高相似度得分，这类情况称为‘半真半假’。在COCO数据集上，CLIP仅40.6%的情况下选择正确且更短的描述，当新增细节为关系时，准确率降至32.9%。我们追溯其根源在于对句子各部分的弱监督：对比学习对完整句子进行对齐，却未显式要求个体实体和关系具有语义锚定。为此提出CS-CLIP（组件监督的CLIP），将描述分解为实体与关系单元，为每个单元构建最小修改的反例，并微调模型使正确单元得分高于其反例，同时保持标准双编码器推理结构。实验表明，该方法将半真半假准确率提升至69.3%，并在多个组合基准上平均性能提升5.7分，表明减少半真半假错误有助于整体组合理解能力提升。代码已开源。
原文摘要 · Abstract (English)
When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。