用多图多标签训练提升CLIP的组合理解能力，兼顾检索性能
Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning
- 通过融合多图像与对应描述进行微调，增强模型对概念关系的理解
- 在SugarCrepe++上实现最佳组合推理表现，同时提升下游检索效果
- 适配不同架构和预训练版本，微调时间短且效果稳定
视觉-语言模型如CLIP在分类和检索任务中展现出强大的零样本能力，但其在组合推理方面仍存在不足——难以理解概念间的关联。最新基准SugarCrepe++显示，以往改进组合性的方法主要提升词汇敏感度，却忽视了语义理解，且常导致下游检索性能下降。本文提出CLIC（Compositionally-aware Learning in CLIP），一种基于多图像及其对应描述的新型微调技术。CLIC在多种模型架构和不同预训练的CLIP模型上均提升了词汇与语义层面的组合理解能力，并实现了稳定的检索性能提升。该方法甚至在最新的CLIPS模型上达到当前最优检索表现。所有模型与代码已公开于https://clic-compositional-clip.github.io
原文摘要 · Abstract (English)
Vision-language models like CLIP have demonstrated remarkable zero-shot capabilities in classification and retrieval. However, these models often struggle with compositional reasoning - the ability to understand the relationships between concepts. A recent benchmark, SugarCrepe++, reveals that previous works on improving compositionality have mainly improved lexical sensitivity but neglected semantic understanding. In addition, downstream retrieval performance often deteriorates, although one would expect that improving compositionality should enhance retrieval. In this work, we introduce CLIC (Compositionally-aware Learning in CLIP), a fine-tuning method based on a novel training technique combining multiple images and their associated captions. CLIC improves compositionality across architectures as well as differently pre-trained CLIP models, both in terms of lexical and semantic understanding, and achieves consistent gains in retrieval performance. This even applies to the recent CLIPS, which achieves SOTA retrieval performance. Nevertheless, the short fine-tuning with CLIC leads to an improvement in retrieval and to the best compositional CLIP model on SugarCrepe++. All our models and code are available at https://clic-compositional-clip.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。