解决CLIP在少样本学习中的类别混淆问题，提升分类准确率。
Logits DeConfusion with CLIP for Few-Shot Learning
- 通过多层级特征融合与残差结构消除类别间混淆
- 在多个数据集上显著提升少样本分类性能
- 适合需要高精度分类的视觉-语言任务应用
尽管CLIP具备强大的视觉-语言对齐能力，在零样本和少样本学习中表现良好，但实验发现其输出的logits存在严重的类别间混淆问题，影响分类准确性。为此，我们提出一种新方法Logits DeConfusion，结合多层级适配器融合（MAF）模块与类别间去混淆（ICD）模块，有效学习并消除logits中的类别混淆。MAF从不同层级提取特征并统一融合，增强表征能力；ICD采用可学习的残差结构，直接在logits层面消除类别混淆。实验结果表明，该方法显著提升了分类性能，缓解了类别混淆问题。代码已开源：https://github.com/LiShuo1001/LDC。
原文摘要 · Abstract (English)
With its powerful visual-language alignment capability, CLIP performs well in zero-shot and few-shot learning tasks. However, we found in experiments that CLIP's logits suffer from serious inter-class confusion problems in downstream tasks, and the ambiguity between categories seriously affects the accuracy. To address this challenge, we propose a novel method called Logits DeConfusion, which effectively learns and eliminates inter-class confusion in logits by combining our Multi-level Adapter Fusion (MAF) module with our Inter-Class Deconfusion (ICD) module. Our MAF extracts features from different levels and fuses them uniformly to enhance feature representation. Our ICD learnably eliminates inter-class confusion in logits with a residual structure. Experimental results show that our method can significantly improve the classification performance and alleviate the inter-class confusion problem. The code is available at https://github.com/LiShuo1001/LDC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。