用大模型零样本提升CLIP分类，不需训练即可跨域适配。
TLAC: Two-stage LMM Augmented CLIP for Zero-Shot Classification
- 借助大模型识别图像物体，再用CLIP文本编码器匹配类别
- 在11个数据集上9次超越现有方法，最高达83.6%准确率
- 完全免训练，适合快速部署到新领域或小样本场景
对比语言-图像预训练（CLIP）在图像分类任务中展现出出色的零样本性能。然而，当前最先进方法通常依赖提示学习或基于适配器的微调技术来优化CLIP表现。这种微调需求显著限制了CLIP在新数据集和新领域的适应性，且每次需耗费大量时间和计算资源。为克服此局限，我们提出简单而有效的免训练方法：单阶段大模型增强CLIP（SLAC）与两阶段大模型增强CLIP（TLAC），利用如Gemini等强大预训练大视觉语言模型（LMM）进行图像分类。该方法通过提示LMM识别图像中的物体，再由CLIP文本编码器根据语义相似性匹配最相关的数据集类别。我们的模型在11个基类到新类数据集上取得了9项领先结果，包括ImageNet、SUN397和Caltech101。TLAC模型整体准确率达83.44%，比此前最先进少样本方法高出6.75%。相较于其他免训练方法，TLAC在13个数据集上平均准确率为83.6%，较之前方法提升9.7%。代码已公开于https://github.com/ans92/TLAC。
原文摘要 · Abstract (English)
Contrastive Language-Image Pretraining (CLIP) has shown impressive zero-shot performance on image classification. However, state-of-the-art methods often rely on fine-tuning techniques like prompt learning and adapter-based tuning to optimize CLIP's performance. The necessity for fine-tuning significantly limits CLIP's adaptability to novel datasets and domains. This requirement mandates substantial time and computational resources for each new dataset. To overcome this limitation, we introduce simple yet effective training-free approaches, Single-stage LMM Augmented CLIP (SLAC) and Two-stage LMM Augmented CLIP (TLAC), that leverages powerful Large Multimodal Models (LMMs), such as Gemini, for image classification. The proposed methods leverages the capabilities of pre-trained LMMs, allowing for seamless adaptation to diverse datasets and domains without the need for additional training. Our approaches involve prompting the LMM to identify objects within an image. Subsequently, the CLIP text encoder determines the image class by identifying the dataset class with the highest semantic similarity to the LLM predicted object. Our models achieved superior accuracy on 9 of 11 base-to-novel datasets, including ImageNet, SUN397, and Caltech101, while maintaining a strictly training-free paradigm. Our TLAC model achieved an overall accuracy of 83.44%, surpassing the previous state-of-the-art few-shot methods by a margin of 6.75%. Compared to other training-free approaches, our TLAC method achieved 83.6% average accuracy across 13 datasets, a 9.7% improvement over the previous methods. Our Code is available at https://github.com/ans92/TLAC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。