用图文描述增强CLIP适配器，提升少样本图像分类效果
IDEA: Image Description Enhanced CLIP-Adapter
- 结合图像视觉特征与文本描述，捕捉细粒度语义信息
- 在11个数据集上达到或超越当前最优性能，训练免费
- 适合少样本学习、多模态适配场景的研究者使用
CLIP在模式识别与计算机视觉中取得显著成功，将其迁移至下游任务（如零样本或少样本分类）是多模态学习的热点。然而，现有研究主要关注文本提示学习或视觉适配器调优，未能充分挖掘图像-文本对之间的互补信息与关联性。本文提出图像描述增强型CLIP适配器（IDEA），通过融合图像视觉特征与文本描述，实现对细粒度特征的捕捉。IDEA是一种无需训练的CLIP适配方法，在多个任务上表现媲美甚至超越现有先进模型。此外，我们提出可训练版IDEA（T-IDEA），引入两个轻量级可学习组件（投影器与可学习潜在空间），进一步提升性能，在11个数据集上达到最先进水平。作为重要贡献，我们利用Llama模型构建完整流水线，为11个数据集生成1,637,795组图像-文本描述对，命名为IMD-11。代码与数据已开源。
原文摘要 · Abstract (English)
CLIP (Contrastive Language-Image Pre-training) has attained great success in pattern recognition and computer vision. Transferring CLIP to downstream tasks (e.g. zero- or few-shot classification) is a hot topic in multimodal learning. However, current studies primarily focus on either prompt learning for text or adapter tuning for vision, without fully exploiting the complementary information and correlations among image-text pairs. In this paper, we propose an Image Description Enhanced CLIP-Adapter (IDEA) method to adapt CLIP to few-shot image classification tasks. This method captures fine-grained features by leveraging both visual features and textual descriptions of images. IDEA is a training-free method for CLIP, and it can be comparable to or even exceeds state-of-the-art models on multiple tasks. Furthermore, we introduce Trainable-IDEA (T-IDEA), which extends IDEA by adding two lightweight learnable components (i.e., a projector and a learnable latent space), further enhancing the model's performance and achieving SOTA results on 11 datasets. As one important contribution, we employ the Llama model and design a comprehensive pipeline to generate textual descriptions for images of 11 datasets, resulting in a total of 1,637,795 image-text pairs, named "IMD-11". Our code and data are released at https://github.com/FourierAI/IDEA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。