让CLIP模型支持任意长度文本描述，突破77 token限制
TULIP: Token-length Upgraded CLIP
- 用相对位置编码替代固定绝对位置编码，提升长序列建模能力
- 在图文检索和文生图任务中，长文本效果显著优于基线模型
- 方法通用性强，可适配多种类似CLIP的视觉语言模型
现有视觉语言模型（如CLIP）受限于固定的绝对位置编码，最大输入长度仅为77个词元，难以处理长文本描述。尽管已有研究尝试扩展长度，但普遍存在对远距离词元关系建模能力不足的问题，且仅支持单一固定长度。为此，我们提出TULIP方法，可将CLIP类模型的文本长度灵活升级至任意长度。通过引入相对位置编码改进架构，并设计两阶段训练流程：首先将原始CLIP文本编码器的知识迁移到带相对位置编码的新结构中，其次增强模型对长文本与图像的对齐能力。实验表明，该方法在跨模态检索和文生图任务中均显著优于基线模型。代码已开源。
原文摘要 · Abstract (English)
We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at https://github.com/ivonajdenkoska/tulip.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。