提出解耦全局局部对齐框架，提升模型对组合概念的理解能力。
Decoupled Global-Local Alignment for Improving Compositional Understanding
- 分离全局与局部对齐机制，用自蒸馏保持预训练知识
- 构建200万高质量负样本描述，显著提升组合理解能力
- 在多个基准上实现3.5%以上性能提升，适合视觉语言建模研究者
对比语言-图像预训练（CLIP）通过对齐图像与文本模态在多个下游任务中取得成功。然而，全局对比学习的特性限制了其对组合概念（如关系和属性）的理解能力。尽管近期研究采用全局难例负样本提升组合理解，但这类方法会显著损害模型的通用能力，因强制将文本负样本与图像在嵌入空间中拉远。为此，本文提出解耦全局-局部对齐（DeGLA）框架，在提升组合理解的同时大幅缓解通用能力损失。为保留模型原有能力，引入自蒸馏机制，在全局对齐中将可学习的图像-文本编码器与基于指数移动平均的冻结教师模型对齐，有效缓解微调过程中的灾难性遗忘。为增强组合理解，首先利用大语言模型的上下文学习能力，构建约200万条跨五类的高质量负样本描述；随后提出图像引导对比损失（IGC）与文本引导对比损失（TGC），强化视觉-语言组合建模。大量实验表明，相比先前最先进方法，DeGLA在VALSE、SugarCrepe和ARO基准上平均提升3.5%；同时在11个数据集的零样本分类任务上平均提升13.0%。代码将开源于https://github.com/xiaoxing2001/DeGLA。
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-training (CLIP) has achieved success on multiple downstream tasks by aligning image and text modalities. However, the nature of global contrastive learning limits CLIP's ability to comprehend compositional concepts, such as relations and attributes. Although recent studies employ global hard negative samples to improve compositional understanding, these methods significantly compromise the model's inherent general capabilities by forcibly distancing textual negative samples from images in the embedding space. To overcome this limitation, we introduce a Decoupled Global-Local Alignment (DeGLA) framework that improves compositional understanding while substantially mitigating losses in general capabilities. To optimize the retention of the model's inherent capabilities, we incorporate a self-distillation mechanism within the global alignment process, aligning the learnable image-text encoder with a frozen teacher model derived from an exponential moving average. Under the constraint of self-distillation, it effectively mitigates the catastrophic forgetting of pretrained knowledge during fine-tuning. To improve compositional understanding, we first leverage the in-context learning capability of Large Language Models (LLMs) to construct about 2M high-quality negative captions across five types. Subsequently, we propose the Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC) loss to enhance vision-language compositionally. Extensive experimental results demonstrate the effectiveness of the DeGLA framework. Compared to previous state-of-the-art methods, DeGLA achieves an average enhancement of 3.5% across the VALSE, SugarCrepe, and ARO benchmarks. Concurrently, it obtains an average performance improvement of 13.0% on zero-shot classification tasks across eleven datasets. Our code will be released at https://github.com/xiaoxing2001/DeGLA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。