让CLIP模型更好理解长文本描述,通过局部对齐提升图文匹配精度。
FAST-GOAL: Fast and Efficient Global-local Object Alignment Learning

- 分区域提取图像局部特征,逐句匹配文本,实现细粒度对齐。
- 在DOCCI和DCI数据集上,图文匹配准确率提升8.7%以上。
- 适合需要精准图文理解的场景,如复杂描述生成与检索。
视觉语言模型如CLIP在对齐图像与文本方面表现优异,但面对长而详细的文本描述时表现不佳,因其预训练使用的是简短标题。本文提出FAST-GOAL(Fast and Efficient Global-local Object Alignment Learning),一种高效微调方法,通过全局-局部语义对齐增强CLIP处理长文本的能力。该方法包含两个核心组件:首先,快速局部图像-句子匹配(FLISM)利用目标检测与空间分割高效提取图像局部区域,并与对应句子进行匹配;其次,基于标记相似性的学习(TSL)最大化特定图像区域的补丁标记与其对应区域嵌入之间的相似性,同样应用于文本,以强化模型捕捉细节对应关系的能力。此外,我们构建了GLIT100k数据集,包含全局图像-长标题对及由上下文推导出的局部配对,其中局部描述源自全局标题,保持语义连贯性。在长标题数据集(DOCCI、DCI)和短标题数据集(MSCOCO、Flickr30k)上的大量实验表明,FAST-GOAL显著优于基线模型,在保持计算效率的同时有效提升CLIP对详细文本描述的适应能力。
原文摘要 · Abstract (English)
Vision-language models such as CLIP have shown impressive capabilities in aligning images and text, but they often struggle with lengthy and detailed text descriptions due to pre-training on short and concise captions. We present FAST-GOAL (Fast and Efficient Global-local Object Alignment Learning), an efficient fine-tuning method that enhances ability of CLIP to handle lengthy text through global-local semantic alignment. Our method consists of two key components. First, Fast Local Image-Sentence Matching (FLISM) efficiently extracts local image regions through object detection and spatial division, then matches them with corresponding sentences. Second, Token Similarity-based Learning (TSL) maximizes the similarity between patch tokens from specific regions in the image and their corresponding region embeddings, applying the same principle to text, which enhances the ability of the model to capture detailed correspondences. Additionally, we introduce GLIT100k, a dataset that provides both global image-lengthy caption pairs and context-derived local pairs, where local descriptions are extracted from global captions to maintain semantic coherence. Through extensive experiments on long caption datasets (DOCCI, DCI) and short caption datasets (MSCOCO, Flickr30k), we demonstrate that FAST-GOAL achieves significant improvements over baselines, enabling effective adaptation of CLIP to detailed textual descriptions while maintaining computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。