用合成描述提升视觉语言模型，仅用部分短文本就能显著提效
CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions
- 只输入部分合成描述给文本编码器，反而比全长度更有效
- 引入自回归生成器模拟重描述过程，提升合成文本质量
- 在MSCOCO等数据集上刷新零样本检索性能，适合改进多模态大模型
先前研究指出，网络爬取的图像-文本对存在噪声，限制了类似CLIP的视觉语言预训练。本文提出一种新框架，通过两种简单但有效的设计，更好地利用丰富描述的合成文本。首先，观察到使用短合成描述时性能远高于长描述，因此仅将部分合成描述输入文本编码器；其次，引入自回归描述生成器，基于配对图像和网络爬取文本，学习预测由先进多模态大模型生成的完整合成描述。实验表明，该框架在跨模态检索任务中显著提升零样本性能，在MSCOCO和Flickr30K上达到新SOTA。此外，训练得到的视觉编码器能增强LLaVA的视觉能力，在多个多模态大模型基准测试中表现明显提升。
原文摘要 · Abstract (English)
Previous works show that noisy, web-crawled image-text pairs may limit vision-language pretraining like CLIP and propose learning with synthetic captions as a promising alternative. Our work continues this effort, introducing two simple yet effective designs to better leverage richly described synthetic captions. Firstly, by observing a strong inverse effect in learning with synthetic captions -- the short synthetic captions can generally lead to MUCH higher performance than full-length ones -- we therefore fed only partial synthetic captions to the text encoder. Secondly, we incorporate an autoregressive captioner to mimic the recaptioning process -- by conditioning on the paired image input and web-crawled text description, the captioner learns to predict the full-length synthetic caption generated by advanced MLLMs. Experiments show that our framework significantly improves zero-shot performance in cross-modal retrieval tasks, setting new SOTA results on MSCOCO and Flickr30K. Moreover, such trained vision encoders can enhance the visual capability of LLaVA, showing strong improvements on a range of MLLM benchmarks. Our project page is https://ucsc-vlaa.github.io/CLIPS/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。