用冻结的单模编码器实现灵活多模态对齐，大幅降低训练成本。
Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment
- 通过选择语义相似的单模编码器，用简单MLP投影对齐视觉与语言
- 在ImageNet上达76%零样本准确率，数据量减少20倍、计算量降65倍
- 适合资源有限但需快速适配多场景的多模态开发人员
近期对比学习的多模态视觉-语言模型（如CLIP）展现出强大的开放世界语义理解能力，已成为视觉-语言应用的标准图像主干网络。然而，最新研究发现高质量训练的单模编码器之间具有高度语义相似性，这引出一个关键问题：能否通过连接单模主干实现视觉-语言任务？为此，我们提出一种新框架，利用冻结的单模编码器进行视觉与语言对齐。该方法包括在隐空间中选择语义相似的编码器，构建富含概念的图像-文本对数据集，并训练简单的MLP投影器。我们在12个零样本分类数据集和2个图像-文本检索数据集上评估了该方法。最佳模型采用DINOv2和All-Roberta-Large文本编码器，在ImageNet上达到76%准确率，相比从头训练的多模态对齐方法，数据需求减少20倍，计算开销降低65倍。该框架提升了多模态模型开发的可及性，支持在多样场景下的灵活适应。代码与构建的数据集已公开于github.com/mayug/freeze-align。
原文摘要 · Abstract (English)
Recent contrastive multimodal vision-language models like CLIP have demonstrated robust open-world semantic understanding, becoming the standard image backbones for vision-language applications. However, recent findings suggest high semantic similarity between well-trained unimodal encoders, which raises a key question: Is there a plausible way to connect unimodal backbones for vision-language tasks? To this end, we propose a novel framework that aligns vision and language using frozen unimodal encoders. It involves selecting semantically similar encoders in the latent space, curating a concept-rich dataset of image-caption pairs, and training simple MLP projectors. We evaluated our approach on 12 zero-shot classification datasets and 2 image-text retrieval datasets. Our best model, utilizing DINOv2 and All-Roberta-Large text encoder, achieves 76\(\%\) accuracy on ImageNet with a 20-fold reduction in data and 65-fold reduction in compute requirements compared multi-modal alignment where models are trained from scratch. The proposed framework enhances the accessibility of multimodal model development while enabling flexible adaptation across diverse scenarios. Code and curated datasets are available at \texttt{github.com/mayug/freeze-align}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。