用段落代替短句做图文检索训练,性能显著提升。
A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

- 用段落替代短句作为文本监督信号,提升长文本检索能力。
- 在DOCCI数据集上比Long-CLIP高14分以上,无需改架构。
- 段落监督让模型更好利用长序列,短句训练超过60词就退化。
对比型图文模型如CLIP和BLIP通常使用短图像描述进行训练,限制了从详细文本中检索图像的能力。尽管Long-CLIP通过位置嵌入插值扩展了词元长度,我们提出一个更简单的问题:文本粒度是否决定长文本检索性能?我们系统研究了从单句描述到多句段落的多种监督方式。基于Qwen2-VL和Llama 3.2 Vision构建合成数据流,为50万张CC3M图像生成多样化的描述、难样本与质量评分段落。为隔离文本粒度影响,仅微调BLIP文本编码器而冻结视觉编码器,共10种配置。段落监督模型在ShareGPT4V上达到Long-CLIP-L水平,在DOCCI上图像到文本检索性能高出14点以上,且未修改模型结构。进一步表明,段落监督可有效利用长序列,而仅用短句训练时性能在超过60词后下降。增加短句多样性对短描述检索有帮助但收益递减;段落监督持续提升长描述与难样本表现,纯文本微调中难样本反而有害。在Flickr30k、COCO、ShareGPT4V和DOCCI上的评估揭示了文本粒度、检索方向与描述长度间的权衡关系。
原文摘要 · Abstract (English)
Contrastive vision-language models such as CLIP and BLIP are typically trained on short image captions, limiting their ability to retrieve images from detailed textual descriptions. While methods such as Long-CLIP extend the token limit through positional embedding interpolation, we ask a simpler question: does training text granularity alone determine long-text retrieval performance? We present a systematic study of supervision ranging from single captions to multi-sentence paragraphs for contrastive image-text retrieval. Using a synthetic pipeline based on Qwen2-VL and Llama 3.2 Vision, we generate diverse captions, hard negatives, and quality-scored paragraphs for 500K CC3M images. To isolate the effect of text granularity, we fine-tune only the BLIP text encoder while keeping the vision encoder frozen across 10 training configurations. Our paragraph-supervised models match Long-CLIP-L on ShareGPT4V and outperform it by more than 14 points on DOCCI for image-to-text retrieval, without architectural changes. We further show that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens. Increasing caption diversity improves short-caption retrieval with diminishing returns, while paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning. Evaluations on Flickr30k, COCO, ShareGPT4V, and DOCCI provide a comprehensive analysis of the trade-offs between text granularity, retrieval direction, and description length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。