用对比描述提升视觉分类,让模型更好区分相似类别。
Enhancing Visual Classification using Comparative Descriptors
- 设计对比描述,突出目标类与其他相似类的差异特征。
- 在CLIP框架中引入对比描述后,零样本分类准确率显著提升。
- 适合需要精细分类的场景,如医疗影像、细粒度图像识别。
视觉语言模型(如CLIP)在视觉分类任务中的表现,可通过融合大语言模型(如GPT)的语义知识得到增强。近期研究表明,在零样本分类任务中,包含额外线索、高层概念甚至随机字符的描述符,常优于仅使用类别名称的描述符。许多分类任务中,虽然顶1准确率相对较低,但顶5准确率明显更高,表明误分类主要集中在少数相似类别之间,反映出模型在细微差别上的区分能力不足。为此,本文提出一种新的对比描述符概念:强调目标类别与其最相似类别之间的独特特征,以增强类别间区分度。通过生成并整合这些对比描述符到分类框架中,我们提升了语义聚焦效果,改善了分类精度。此外,引入筛选机制使描述符更贴近图像嵌入在CLIP空间中的位置,进一步提升性能。实验表明,该方法有效提升了视觉分类任务的准确性与鲁棒性,尤其针对细微类间差异的挑战。
原文摘要 · Abstract (English)
The performance of vision-language models (VLMs), such as CLIP, in visual classification tasks, has been enhanced by leveraging semantic knowledge from large language models (LLMs), including GPT. Recent studies have shown that in zero-shot classification tasks, descriptors incorporating additional cues, high-level concepts, or even random characters often outperform those using only the category name. In many classification tasks, while the top-1 accuracy may be relatively low, the top-5 accuracy is often significantly higher. This gap implies that most misclassifications occur among a few similar classes, highlighting the model's difficulty in distinguishing between classes with subtle differences. To address this challenge, we introduce a novel concept of comparative descriptors. These descriptors emphasize the unique features of a target class against its most similar classes, enhancing differentiation. By generating and integrating these comparative descriptors into the classification framework, we refine the semantic focus and improve classification accuracy. An additional filtering process ensures that these descriptors are closer to the image embeddings in the CLIP space, further enhancing performance. Our approach demonstrates improved accuracy and robustness in visual classification tasks by addressing the specific challenge of subtle inter-class differences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。