arXiv:2602.22419cs.CV2026-02被引 1

CLIP模型过度关注标题句,新方法通过去偏训练提升长文本理解能力。

CLIP Is Shortsighted: Paying Attention Beyond the First Sentence

  • 训练时移除摘要句,用分句采样和填充分布监督信号
  • 长文本检索性能超越现有模型,短文本也更鲁棒
  • 无需额外参数,可直接替换Long-CLIP使用

CLIP模型通过互联网规模的图文对比学习获得可迁移的多模态特征,广泛用于零样本分类、跨模态检索、文生图扩散及大视觉语言模型的图像编码。然而其预训练主要基于短标题配图,导致模型偏向编码显著对象的简单描述,对复杂场景和密集描述的对齐能力较弱。尽管近期工作通过在小规模长标题数据集上微调缓解此问题,我们发现人类与LLM生成的长标题通常以一句摘要开头,随后是详细描述——这种结构在训练中形成捷径,使模型注意力集中于开头句子和早期词元,削弱了对后续内容的对齐。为此,我们提出DeBias-CLIP:训练时移除摘要句,并采用分句采样与文本词元填充,使监督信号均匀分布于所有词元位置。DeBias-CLIP在长文本检索上达到当前最优性能,同时提升短文本检索效果,且对句子顺序变化不敏感。它作为Long-CLIP的即插即用替代品,无需额外可训练参数。

原文摘要 · Abstract (English)

CLIP models learn transferable multi-modal features via image-text contrastive learning on internet-scale data. They are widely used in zero-shot classification, multi-modal retrieval, text-to-image diffusion, and as image encoders in large vision-language models. However, CLIP's pretraining is dominated by images paired with short captions, biasing the model toward encoding simple descriptions of salient objects and leading to coarse alignment on complex scenes and dense descriptions. While recent work mitigates this by fine-tuning on small-scale long-caption datasets, we identify an important common bias: both human- and LLM-generated long captions typically begin with a one-sentence summary followed by a detailed description. We show that this acts as a shortcut during training, concentrating attention on the opening sentence and early tokens and weakening alignment over the rest of the caption. To resolve this, we introduce DeBias-CLIP, which removes the summary sentence during training and applies sentence sub-sampling and text token padding to distribute supervision across all token positions. DeBias-CLIP achieves state-of-the-art long-text retrieval, improves short-text retrieval, and is less sensitive to sentence order permutations. It is a drop-in replacement for Long-CLIP with no additional trainable parameters.

多模态CLIP文本对齐去偏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。