通过重建与对齐文本描述,提升CLIP的组合推理能力。
Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions
- 引入词级重构与句级对齐双重辅助目标,增强文本关系理解。
- 在5个基准上达到最优,最高超越基线4.1%。
- 适用于需精准理解图文关系的视觉语言任务。
尽管近期取得进展,基于标准对比学习的目标训练的视觉语言模型仍难以实现组合推理——即理解视觉与语言元素之间的结构化关系。这一缺陷主要源于文本编码器倾向于关注单个词汇而非其关联性,而对比学习又强化了词汇与视觉对象的直接对齐。本文提出READ(REconstruction and Alignment of text Descriptions)微调方法,在对比学习基础上添加两项辅助目标:(1) 词级重构目标,利用冻结的预训练解码器根据原始标题嵌入重建替代性描述;(2) 句级对齐目标,显式对齐语义相同但表述不同的句子在嵌入空间中的表示。实验表明,将READ应用于预训练的CLIP模型所得到的READ-CLIP,在五个主流组合推理基准上均达到当前最优性能,相较于最强传统微调基线最高提升4.1%。此外,该方法同样有效提升现有CLIP变体(如NegCLIP和FSC-CLIP)的表现。定量与定性分析显示,重构与对齐目标具有互补作用:前者促使编码器捕捉句内词汇间的关系,后者确保不同表达方式的同义句保持一致的表示。
原文摘要 · Abstract (English)
Despite recent advances, vision-language models trained with standard contrastive objectives still struggle with compositional reasoning -- the ability to understand structured relationships between visual and linguistic elements. This shortcoming is largely due to the tendency of the text encoder to focus on individual words rather than their relations, a limitation reinforced by contrastive training that primarily aligns words with visual objects. In this paper, we introduce REconstruction and Alignment of text Descriptions (READ), a fine-tuning method designed to enhance compositional reasoning by adding two auxiliary objectives to the contrastive learning: (1) a token-level reconstruction objective, where a frozen pre-trained decoder reconstructs alternative captions based on the embedding of the original caption; and (2) a sentence-level alignment objective, which explicitly aligns paraphrased sentences in the embedding space. We show that READ-CLIP, a model derived by applying the READ method to the pre-trained CLIP model, achieves the state-of-the-art performance across five major compositional reasoning benchmarks, outperforming the strongest conventional fine-tuning baseline by up to 4.1%. Furthermore, applying the READ to existing CLIP variants (including NegCLIP and FSC-CLIP) also improves performance on these benchmarks. Quantitative and qualitative analyses reveal that our proposed objectives -- reconstruction and alignment -- offer complementary benefits: the former encourages the encoder to capture relationships between words within a caption, while the latter ensures consistent representations for paraphrases expressed with different wording.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。