arXiv:2409.11338cs.CV2024-09被引 3

通过降低图像嵌入的内部重叠,提升CLIP少样本分类性能。

CLIP Adaptation by Intra-modal Overlap Reduction

  • 在图像空间中训练轻量适配器,减少同类样本的嵌入重叠。
  • 在多个标准数据集上准确率提升,且对分布偏移更鲁棒。
  • 适合追求高效少样本分类的模型优化研究者。

许多方法被提出用于将预训练的CLIP模型适应于少样本分类任务。由于在大规模语料上训练,CLIP在适应少样本分类时表现出良好泛化能力。本文分析了图像空间中嵌入表示的模内重叠问题。研究表明,由于对比学习机制,CLIP模型的嵌入在图像空间中成对与非成对样本间存在高余弦相似度分布重叠,这影响了依赖图像空间相似性的无训练少样本分类方法的表现。为此,我们提出在Google Open Images数据集的通用样本上训练一个轻量级适配器,以减少模内重叠,从而提升少样本无训练分类的准确性。通过大量实证分析验证,减少模内重叠可带来:a)多个标准数据集上的性能提升;b)对分布偏移更强的鲁棒性;c)更高的特征方差,使特征更具判别力,适用于下游任务。

原文摘要 · Abstract (English)

Numerous methods have been proposed to adapt a pre-trained foundational CLIP model for few-shot classification. As CLIP is trained on a large corpus, it generalises well through adaptation to few-shot classification. In this work, we analyse the intra-modal overlap in image space in terms of embedding representation. Our analysis shows that, due to contrastive learning, embeddings from CLIP model exhibit high cosine similarity distribution overlap in the image space between paired and unpaired examples affecting the performance of few-shot training-free classification methods which rely on similarity in the image space for their predictions. To tackle intra-modal overlap we propose to train a lightweight adapter on a generic set of samples from the Google Open Images dataset demonstrating that this improves accuracy for few-shot training-free classification. We validate our contribution through extensive empirical analysis and demonstrate that reducing the intra-modal overlap leads to a) improved performance on a number of standard datasets, b) increased robustness to distribution shift and c) higher feature variance rendering the features more discriminative for downstream tasks.

CLIP少样本学习嵌入优化特征判别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。