提升视觉语言模型的组合理解能力，同时不损失多模态性能。
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality
- 引入局部硬负样本损失与选择性校准正则化，实现细粒度负样本监督。
- 在组合性任务上达到顶尖水平，零样本多模态任务性能保持不变。
- 适合需要兼顾推理能力与多模态泛化性的研究者使用。
本文提出一种新方法，旨在增强预训练视觉语言模型（VLMs）的组合理解能力，同时不牺牲其在零样本多模态任务中的表现。传统微调方法常以降低多模态能力为代价提升组合推理，主要源于全局硬负样本（HN）损失——该损失对比图像与文本的全局表示，导致与原文本高度相似的硬负样本被过度拉远，破坏模型多模态表征。为此，我们提出细粒度选择性校准CLIP（FSC-CLIP），融合局部硬负样本损失与选择性校准正则化，实现细粒度负样本监督并保护模型表征完整性。在多个组合性与多模态任务基准上的广泛评估表明，FSC-CLIP不仅在组合性任务上达到当前最优水平，且保持了强大的零样本多模态能力。代码已开源：https://github.com/ytaek-oh/fsc-clip。
原文摘要 · Abstract (English)
In this paper, we propose a new method to enhance compositional understanding in pre-trained vision and language models (VLMs) without sacrificing performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches often improve compositional reasoning at the cost of degrading multi-modal capabilities, primarily due to the use of global hard negative (HN) loss, which contrasts global representations of images and texts. This global HN loss pushes HN texts that are highly similar to the original ones, damaging the model's multi-modal representations. To overcome this limitation, we propose Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard negative loss and selective calibrated regularization. These innovations provide fine-grained negative supervision while preserving the model's representational integrity. Our extensive evaluations across diverse benchmarks for both compositionality and multi-modal tasks show that FSC-CLIP not only achieves compositionality on par with state-of-the-art models but also retains strong multi-modal capabilities. Code is available at: https://github.com/ytaek-oh/fsc-clip.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。