利用层级提示让CLIP在零样本分类中犯更合理的错误
Making Better Mistakes in CLIP-Based Zero-Shot Classification with Hierarchy-Aware Language Prompts
- 用大语言模型生成带层级信息的文本提示
- 在5个不同规模数据集上均优于现有方法
- 适合需要可控误判场景的零样本分类应用
近期研究利用在海量网络文本上训练的大语言模型（LLMs），为下游类别生成文本描述，以提升基于CLIP的零样本图像分类性能。尽管多数方法聚焦于提高准确率，本文关注‘犯更合理的错误’——即根据下游任务的标签层级结构定义错误严重性。由于CLIP的图像编码器通过语言监督信号训练，其隐式捕捉了类别间的层次语义关系，因此天然适合此类目标。我们提出HAPrompts方法，通过调用语言模型生成特定类别的文本表示，作为CLIP的零样本分类器进行图像分类。据我们所知，这是首个在基于CLIP的零样本分类中引入‘合理犯错’理念的工作。实验在五个具有不同层级高度的数据集上验证，本方法在整体表现上超越相关方法。代码与自动生成的图像提示已开源。
原文摘要 · Abstract (English)
Recent studies are leveraging advancements in large language models (LLMs) trained on extensive internet-crawled text data to generate textual descriptions of downstream classes in CLIP-based zero-shot image classification. While most of these approaches aim at improving accuracy, our work focuses on ``making better mistakes", of which the mistakes' severities are derived from the given label hierarchy of downstream tasks. Since CLIP's image encoder is trained with language supervising signals, it implicitly captures the hierarchical semantic relationships between different classes. This motivates our goal of making better mistakes in zero-shot classification, a task for which CLIP is naturally well-suited. Our approach (HAPrompts) queries the language model to produce textual representations for given classes as zero-shot classifiers of CLIP to perform image classification on downstream tasks. To our knowledge, this is the first work to introduce making better mistakes in CLIP-based zero-shot classification. Our approach outperforms the related methods in a holistic comparison across five datasets of varying scales with label hierarchies of different heights in our experiments. Our code and LLM-generated image prompts: \href{https://github.com/ltong1130ztr/HAPrompts}{https://github.com/ltong1130ztr/HAPrompts}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。