用跨模态注意力融合图文特征，提升CLIP模型泛化能力
Multi-Modal Adapter for Vision-Language Models
- 引入可训练的多头注意力层，联合建模图文特征交互
- 在未见类别上表现优于现有适配方法，泛化能力更强
- 适合希望低成本提升视觉语言模型性能的研究者
大型预训练视觉语言模型（如CLIP）在多种图像分类任务中表现出色，无需微调。少样本CLIP的表现已接近专为下游任务训练的模型。近期研究显示，轻量级适配方法可进一步提升CLIP性能。然而，已有方法对视觉与文本模态分别进行适配，忽略了两者间的交互关系。本文提出多模态适配器（Multi-Modal Adapter），通过添加一个可训练的多头注意力层，融合文本与图像特征，实现对双模态的联合增量适配。该方法在未见类别上的表现优于现有适配策略，展现出更强的泛化能力。我们还进行了额外消融实验和分析，验证并解释了该方法的有效性。
原文摘要 · Abstract (English)
Large pre-trained vision-language models, such as CLIP, have demonstrated state-of-the-art performance across a wide range of image classification tasks, without requiring retraining. Few-shot CLIP is competitive with existing specialized architectures that were trained on the downstream tasks. Recent research demonstrates that the performance of CLIP can be further improved using lightweight adaptation approaches. However, previous methods adapt different modalities of the CLIP model individually, ignoring the interactions and relationships between visual and textual representations. In this work, we propose Multi-Modal Adapter, an approach for Multi-Modal adaptation of CLIP. Specifically, we add a trainable Multi-Head Attention layer that combines text and image features to produce an additive adaptation of both. Multi-Modal Adapter demonstrates improved generalizability, based on its performance on unseen classes compared to existing adaptation methods. We perform additional ablations and investigations to validate and interpret the proposed approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。