用冻结大模型做文本编码，让图文预训练更省数据、支持长文本和多语言。
FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training
- 用冻结大模型替代传统文本编码器，天然处理长文本输入。
- 在CC3M上图像分类准确率比前人高4.9%，跨语言图文检索提升超30%。
- 适合追求多语言性能、长文本理解的图文模型研究者使用。
语言-图像预训练因特定格式数据有限及文本编码器能力受限而面临挑战。现有方法通过数据增强和架构改进试图解决，但仍难以处理长文本，且传统CLIP文本编码器的固有局限导致下游泛化效果不佳。本文提出FLAME（Frozen Large Language Models Enable data-efficient language-image pre-training），采用冻结的大语言模型作为文本编码器，自然处理长文本输入，并展现出出色的多语言泛化能力。FLAME包含两个核心组件：1）多维度提示蒸馏技术，从长描述中提取多样化的语义表示，更好匹配图像的多面性；2）分面解耦注意力机制，配合离线嵌入策略，确保高效计算。大量实验证明，FLAME性能卓越：在CC3M上训练时，图像分类Top-1准确率比此前最优方法高出4.9%；在YFCC15M上，跨36种语言的平均图像到文本召回率@1相比WIT-400M训练的CLIP提升44.4%，在Urban-1k长文本检索任务中，文本到图像召回率@1提升34.6%。代码已开源。
原文摘要 · Abstract (English)
Language-image pre-training faces significant challenges due to limited data in specific formats and the constrained capacities of text encoders. While prevailing methods attempt to address these issues through data augmentation and architecture modifications, they continue to struggle with processing long-form text inputs, and the inherent limitations of traditional CLIP text encoders lead to suboptimal downstream generalization. In this paper, we propose FLAME (Frozen Large lAnguage Models Enable data-efficient language-image pre-training) that leverages frozen large language models as text encoders, naturally processing long text inputs and demonstrating impressive multilingual generalization. FLAME comprises two key components: 1) a multifaceted prompt distillation technique for extracting diverse semantic representations from long captions, which better aligns with the multifaceted nature of images, and 2) a facet-decoupled attention mechanism, complemented by an offline embedding strategy, to ensure efficient computation. Extensive empirical evaluations demonstrate FLAME's superior performance. When trained on CC3M, FLAME surpasses the previous state-of-the-art by 4.9% in ImageNet top-1 accuracy. On YFCC15M, FLAME surpasses the WIT-400M-trained CLIP by 44.4\% in average image-to-text recall@1 across 36 languages, and by 34.6% in text-to-image recall@1 for long-context retrieval on Urban-1k. Code is available at https://github.com/MIV-XJTU/FLAME.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。