让CLIP理解长文本，通过合成标题提升图文对齐能力
FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Text
- 双分支训练对齐短/长文本与图像，保持短文本性能
- 用可学习区域提示提取局部特征，支持长文本理解
- 适合需要长文本输入的扩散模型等下游应用
CLIP在短文本任务中表现出色，但受限于文本编码器输入长度，难以处理超过77个词的长文本。为此，我们提出FIX-CLIP，包含三个新模块：(1) 双分支训练管道，分别对齐短文本与掩码图像、长文本与原始图像，增强长文本表征并保留短文本能力；(2) 在Transformer层中使用可学习区域提示与单向掩码，实现区域信息提取；(3) 在中间编码层引入层次化特征对齐模块，提升多尺度特征一致性。此外，我们收集了3000万张图像，并利用现有多模态大模型生成长文本描述用于训练。大量实验表明，FIX-CLIP在长文本和短文本检索基准上均达到领先性能。在下游应用中，其文本编码器可直接适配长文本输入的扩散模型，表现优异。代码已开源。
原文摘要 · Abstract (English)
CLIP has shown promising performance across many short-text tasks in a zero-shot manner. However, limited by the input length of the text encoder, CLIP struggles on under-stream tasks with long-text inputs ($>77$ tokens). To remedy this issue, we propose FIX-CLIP, which includes three novel modules: (1) A dual-branch training pipeline that aligns short and long texts with masked and raw images, respectively, which boosts the long-text representation while preserving the short-text ability. (2) Multiple learnable regional prompts with unidirectional masks in Transformer layers for regional information extraction. (3) A hierarchical feature alignment module in the intermediate encoder layers to promote the consistency of multi-scale features. Furthermore, we collect 30M images and utilize existing MLLMs to synthesize long-text captions for training. Extensive experiments show that FIX-CLIP achieves state-of-the-art performance on both long-text and short-text retrieval benchmarks. For downstream applications, we reveal that FIX-CLIP's text encoder delivers promising performance in a plug-and-play manner for diffusion models with long-text input. The code is available at https://github.com/bcwang-sjtu/Fix-CLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。