arXiv:2504.01916cs.CVcs.AI2025-04CVPR被引 31

让CLIP能处理更长文本,提升图文精细对齐能力

FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs

  • 通过扩展位置编码和动态聚合实现长文本对齐
  • 在零样本检索和文生图任务上超越现有方法
  • 适合需要细粒度图文理解的场景

作为开创性的视觉语言模型,CLIP(对比语言-图像预训练)在多个领域和广泛下游任务中取得了显著成功。然而,主流CLIP模型中的文本编码器仅能处理77个文本标记,限制了其对更长、细节丰富的描述的有效处理能力。此外,CLIP模型常难以有效捕捉详细的视觉与文本信息,影响其在需要细粒度分析任务上的表现。为此,我们提出一种新方法FineLIP,通过在CLIP框架内引入细粒度对齐与更长文本输入,拓展了CLIP的能力。FineLIP首先扩展位置编码以支持更长文本,随后动态聚合局部图像与文本标记,再利用聚合结果实现细粒度的标记级跨模态对齐。我们在包含长而详细的描述数据集上验证了该模型,在零样本跨模态检索与文本到图像生成两个任务中均取得优异表现,优于现有最先进方法。全面的消融实验也证实了关键设计元素的有效性。

原文摘要 · Abstract (English)

As a pioneering vision-language model, CLIP (Contrastive Language-Image Pre-training) has achieved significant success across various domains and a wide range of downstream vision-language tasks. However, the text encoders in popular CLIP models are limited to processing only 77 text tokens, which constrains their ability to effectively handle longer, detail-rich captions. Additionally, CLIP models often struggle to effectively capture detailed visual and textual information, which hampers their performance on tasks that require fine-grained analysis. To address these limitations, we present a novel approach, \textbf{FineLIP}, that extends the capabilities of CLIP. FineLIP enhances cross-modal text-image mapping by incorporating \textbf{Fine}-grained alignment with \textbf{L}onger text input within the CL\textbf{IP}-style framework. FineLIP first extends the positional embeddings to handle longer text, followed by the dynamic aggregation of local image and text tokens. The aggregated results are then used to enforce fine-grained token-to-token cross-modal alignment. We validate our model on datasets with long, detailed captions across two tasks: zero-shot cross-modal retrieval and text-to-image generation. Quantitative and qualitative experimental results demonstrate the effectiveness of FineLIP, outperforming existing state-of-the-art approaches. Furthermore, comprehensive ablation studies validate the benefits of key design elements within FineLIP.

视觉语言CLIP改进细粒度对齐长文本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。