用CLIP处理模糊标签，让模型在无标注情况下也能准确识别新类别。
CLIP-driven Zero-shot Learning with Ambiguous Labels
- 基于CLIP提取图像与标签特征，通过语义挖掘融合生成判别性标签嵌入。
- 设计部分零样本损失函数，动态加权候选标签并减少语义错配。
- 适合标签不准确的现实场景，尤其对弱监督或噪声数据集有显著提升。
零样本学习（ZSL）旨在利用已知类别的语义信息识别未见类别，但现有方法通常假设训练样本具有精确标签。然而在真实场景中，标签噪声和模糊性会显著降低ZSL性能。为此，我们提出一种新的CLIP驱动的局部标签零样本学习框架（CLIP-PZSL），以应对标签模糊问题。首先，利用CLIP提取实例和标签特征；随后，通过语义挖掘模块融合两者，生成判别性标签嵌入。我们还引入部分零样本损失，根据候选标签与实例的相关性分配权重，并对齐实例与标签嵌入以最小化语义偏差。随着训练进行，真实标签逐渐被识别，优化后的标签与嵌入进一步提升特征对齐效果。在多个数据集上的全面实验表明，CLIP-PZSL具有明显优势。
原文摘要 · Abstract (English)
Zero-shot learning (ZSL) aims to recognize unseen classes by leveraging semantic information from seen classes, but most existing methods assume accurate class labels for training instances. However, in real-world scenarios, noise and ambiguous labels can significantly reduce the performance of ZSL. To address this, we propose a new CLIP-driven partial label zero-shot learning (CLIP-PZSL) framework to handle label ambiguity. First, we use CLIP to extract instance and label features. Then, a semantic mining block fuses these features to extract discriminative label embeddings. We also introduce a partial zero-shot loss, which assigns weights to candidate labels based on their relevance to the instance and aligns instance and label embeddings to minimize semantic mismatch. As the training goes on, the ground-truth labels are progressively identified, and the refined labels and label embeddings in turn help improve the semantic alignment of instance and label features. Comprehensive experiments on several datasets demonstrate the advantage of CLIP-PZSL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。