用提示词增强和加权训练，提升CLIP在噪声标签下的少样本学习能力。
CRoF: CLIP-based Robust Few-shot Learning on Noisy Labels
- 设计任务导向提示生成器，增强类别描述的区分度。
- 在不同噪声比例下，性能超越微调和原始CLIP模型。
- 适合处理标签不准确的少样本图像分类任务。
噪声标签会因新领域特征不准确而威胁少样本学习（FSL）的鲁棒性。尽管大模型CLIP在图像-文本嵌入相似性上表现优异，但对噪声标签导致的误分类仍敏感。本文提出一种新方法CRoF，旨在提升CLIP在噪声数据下的域泛化能力。该方法为基于CLIP的模型设计了通用插件模块：通过任务导向提示生成器，提供更具区分性的类别描述，扩大类间文本嵌入距离；不完全依赖零样本分类，采用类似标签平滑的加权策略，在新领域噪声少样本数据上微调CLIP，权重综合考虑CLIP先验知识与原始标签信息，确保可靠性。此外，多标签损失函数进一步支持此范式下的稳健训练。大量实验表明，作为插件，CRoF在多种噪声类型和噪声比例下均优于微调及原始CLIP模型。
原文摘要 · Abstract (English)
Noisy labels threaten the robustness of few-shot learning (FSL) due to the inexact features in a new domain. CLIP, a large-scale vision-language model, performs well in FSL on image-text embedding similarities, but it is susceptible to misclassification caused by noisy labels. How to enhance domain generalization of CLIP on noisy data within FSL tasks is a critical challenge. In this paper, we provide a novel view to mitigate the influence of noisy labels, CLIP-based Robust Few-shot learning (CRoF). CRoF is a general plug-in module for CLIP-based models. To avoid misclassification and confused label embedding, we design the few-shot task-oriented prompt generator to give more discriminative descriptions of each category. The proposed prompt achieves larger distances of inter-class textual embedding. Furthermore, rather than fully trusting zero-shot classification by CLIP, we fine-tune CLIP on noisy few-shot data in a new domain with a weighting strategy like label-smooth. The weights for multiple potentially correct labels consider the relationship between CLIP's prior knowledge and original label information to ensure reliability. Our multiple label loss function further supports robust training under this paradigm. Comprehensive experiments show that CRoF, as a plug-in, outperforms fine-tuned and vanilla CLIP models on different noise types and noise ratios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。