用冻结模型+轻量适配器，低成本实现顶尖图文理解能力
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- 用冻结的图像和语言模型，加轻量查询变换器桥接视觉与语言
- 零样本下在VQAv2上比Flamingo80B高8.7%，参数少54倍
- 能零样本根据自然语言指令生成图像描述，适合高效多模态应用
由于大规模模型的端到端训练成本高昂，视觉-语言预训练变得越来越昂贵。本文提出BLIP-2，一种通用且高效的预训练策略，通过使用现成的冻结图像编码器和冻结大语言模型来启动视觉-语言预训练。BLIP-2采用轻量级查询变换器弥合模态差距，分两阶段预训练：第一阶段从冻结图像编码器中启动视觉-语言表征学习；第二阶段从冻结语言模型中启动视觉到语言的生成学习。尽管可训练参数远少于现有方法，BLIP-2在多种视觉-语言任务上达到领先性能。例如，在零样本VQAv2任务上，其表现比Flamingo80B高出8.7%，但可训练参数仅为其1/54。我们还展示了该模型在零样本图像到文本生成方面展现的新兴能力，能够遵循自然语言指令进行生成。
原文摘要 · Abstract (English)
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。