让CLIP理解长文本描述，通过局部与全局语义对齐提升图文匹配能力。
GOAL: Global-local Object Alignment Learning
- 引入局部图像-句子匹配与词元相似性传播机制，增强细粒度对齐。
- 在三个新基准上显著优于基线，长文本检索准确率提升明显。
- 适合需要精细理解复杂文本描述的视觉语言任务使用。
视觉语言模型如CLIP在对齐图像与文本方面表现优异，但其训练侧重于简短描述，难以处理长且详细的文本。本文提出GOAL（Global-local Object Alignment Learning），一种新型微调方法，通过结合图像与长文本间的全局和局部语义对齐，增强CLIP对长文本的理解能力。方法包含两个关键组件：局部图像-句子匹配（LISM），用于识别图像片段与描述句子间的对应关系；以及基于词元相似性的学习（TSL），通过匹配对高效传播局部元素注意力。在三个针对图像-长文本检索的新基准上评估，结果表明GOAL显著优于基线CLIP微调，验证了该方法在适配长文本描述上的有效性。大量实验显示，局部语义对齐与全局上下文结合可生成更细致、更具代表性的嵌入表示，尤其适用于需细粒度理解长文本的任务。
原文摘要 · Abstract (English)
Vision-language models like CLIP have shown impressive capabilities in aligning images and text, but they often struggle with lengthy and detailed text descriptions because of their training focus on short and concise captions. We present GOAL (Global-local Object Alignment Learning), a novel fine-tuning method that enhances CLIP's ability to handle lengthy text by leveraging both global and local semantic alignments between image and lengthy text. Our approach consists of two key components: Local Image-Sentence Matching (LISM), which identifies corresponding pairs between image segments and descriptive sentences, and Token Similarity-based Learning (TSL), which efficiently propagates local element attention through these matched pairs. Evaluating GOAL on three new benchmarks for image-lengthy text retrieval, we demonstrate significant improvements over baseline CLIP fine-tuning, establishing a simple yet effective approach for adapting CLIP to detailed textual descriptions. Through extensive experiments, we show that our method's focus on local semantic alignment alongside global context leads to more nuanced and representative embeddings, particularly beneficial for tasks requiring fine-grained understanding of lengthy text descriptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。