无需标注数据,靠自学习实现零样本图像分类
No Labels Needed: Zero-Shot Image Classification with Collaborative Self-Learning
- 用视觉语言模型与预训练视觉模型协同生成伪标签
- 在无标注数据下准确率超越基线方法
- 适合缺乏标注数据的场景,不依赖大语言模型
尽管深度学习(如CNN和ViT)显著提升了分类性能,但其对大量标注数据的依赖在实际应用中成为障碍。本文提出一种新型零样本图像分类框架,结合视觉语言模型(VLM)与预训练视觉模型,在自学习循环中运行。仅需类别名称,无需任何标注训练数据,通过置信度驱动的伪标签策略,在测试数据上直接训练轻量级分类器,实现动态适应。VLM识别高置信度样本,预训练视觉模型增强其视觉表征,这些增强特征迭代训练分类器,使系统在无监督条件下捕捉互补的语义与视觉线索。值得注意的是,该方法避免了VLM微调和大型语言模型使用,仅依赖视觉模型降低对语义表征的依赖。在十个不同数据集上的实验表明,该方法优于基线零样本方法。
原文摘要 · Abstract (English)
While deep learning, including Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), has significantly advanced classification performance, its typical reliance on extensive annotated datasets presents a major obstacle in many practical scenarios where such data is scarce. Vision-language models (VLMs) and transfer learning with pre-trained visual models appear as promising techniques to deal with this problem. This paper proposes a novel zero-shot image classification framework that combines a VLM and a pre-trained visual model within a self-learning cycle. Requiring only the set of class names and no labeled training data, our method utilizes a confidence-based pseudo-labeling strategy to train a lightweight classifier directly on the test data, enabling dynamic adaptation. The VLM identifies high-confidence samples, and the pre-trained visual model enhances their visual representations. These enhanced features then iteratively train the classifier, allowing the system to capture complementary semantic and visual cues without supervision. Notably, our approach avoids VLM fine-tuning and the use of large language models, relying on the visual-only model to reduce the dependence on semantic representation. Experimental evaluations on ten diverse datasets demonstrate that our approach outperforms the baseline zero-shot method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。