让图文相互指导，提升零训练少样本分类效果
Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP
- 设计图文双向引导机制，利用图像优化文本提示，用文本修正图像匹配异常
- 在多个数据集上超越当前最优零训练方法，部分任务性能提升显著
- 适合追求高效少样本分类且无训练资源的研究者使用
对比语言-图像预训练（CLIP）已在视觉任务中广泛应用，尤其在少样本学习（FSL）中表现优异。然而，现有无需额外训练的CLIP方法多独立处理模态，导致两大问题：图像模态存在严重异常匹配，生成的文本提示质量不一。为此，本文提出互导优化框架TIMO，引入图像引导文本（IGT）组件，通过图像表征修正文本提示质量；同时引入文本引导图像（TGI）组件，借助文本表征缓解图像异常匹配。通过整合IGT与TGI，实现图文双向协同优化。大量实验表明，TIMO显著优于当前最优零训练方法。进一步提出的增强版TIMO-S，在约100倍更少时间成本下，性能超越最佳需训练方法0.33%。代码已公开。
原文摘要 · Abstract (English)
Contrastive Language-Image Pretraining (CLIP) has been widely used in vision tasks. Notably, CLIP has demonstrated promising performance in few-shot learning (FSL). However, existing CLIP-based methods in training-free FSL (i.e., without the requirement of additional training) mainly learn different modalities independently, leading to two essential issues: 1) severe anomalous match in image modality; 2) varying quality of generated text prompts. To address these issues, we build a mutual guidance mechanism, that introduces an Image-Guided-Text (IGT) component to rectify varying quality of text prompts through image representations, and a Text-Guided-Image (TGI) component to mitigate the anomalous match of image modality through text representations. By integrating IGT and TGI, we adopt a perspective of Text-Image Mutual guidance Optimization, proposing TIMO. Extensive experiments show that TIMO significantly outperforms the state-of-the-art (SOTA) training-free method. Additionally, by exploring the extent of mutual guidance, we propose an enhanced variant, TIMO-S, which even surpasses the best training-required methods by 0.33% with approximately 100 times less time cost. Our code is available at https://github.com/lyymuwu/TIMO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。