无需标签,用CLIP评分图像局部区域,提升细粒度分类效果
CLIP-Guided Label-Free Discriminative Region Scoring for Fine-Grained Classification

- 基于CLIP的三种无标签区域评分策略,结合SAM掩码与随机裁剪
- 软负边距评分法表现最佳,伪标签接近真实标签性能
- 随机裁剪比SAM掩码更稳定,尤其在标签噪声下表现更优
近期视觉模型如CLIP和SAM实现了无需训练的分割与语义编码,用于细粒度分类。常见方法是将分割区域的表示与对应标签文本嵌入进行对比。然而,局部区域差异及CLIP评分策略如何影响判别性证据的选择仍不明确,尤其在无真值标签情况下。本文提出统一的CLIP引导无标签区域评分框架,评估基于余弦相似度、边距和熵的评分策略,使用SAM生成掩码和随机裁剪两种区域生成方式,并引入基于全局图像嵌入与局部区域嵌入的两种无标签伪标签变体。在五个细粒度分类数据集上系统比较不同区域生成方法与评分策略。结果表明,软负边距评分表现最强,伪标签评分接近真实标签性能。尽管SAM生成语义合理掩码,但随机裁剪生成的伪标签在所有数据集上均优于SAM,说明随机裁剪保留了更多上下文信息,在伪标签噪声下更稳定。此外,SAM掩码通过聚合所有区域嵌入获益,而随机裁剪在较小top-k子集上表现更佳。这些发现为细粒度分类提供了新视角。
原文摘要 · Abstract (English)
Recent vision models such as CLIP and SAM enable training-free segmentation and semantic encoding for fine-grained classification. A common approach is to compare the representations of segmented image regions with the text prompt embeddings of the corresponding labels. However, it remains unclear how different local regions and CLIP-based scoring strategies affect the selection of discriminative evidence, especially when ground-truth labels are unavailable. In this paper, we propose a unified CLIP-guided label-free region scoring framework for fine-grained classification. The framework evaluates cosine similarity-based, margin-based, and entropy-based scoring strategies using both SAM-generated masks and random crops, and introduces two label-free pseudo-label variants based on global image embeddings and local region embeddings. We conduct experiments on five fine-grained classification datasets to systematically compare different region generation methods and scoring strategies. The results show that Soft Negative Margin scoring achieves the strongest performance, and pseudo-label scoring closely approximates true-label performance. Although SAM produces semantically meaningful masks, random-crop-based pseudo-label scoring consistently outperforms SAM-based scoring across all datasets, suggesting that random crops preserve surrounding information and provide more stable semantic context when pseudo-labels are noisy. In addition, SAM masks benefit from aggregating embeddings from all regions, whereas random crops tend to perform better with a smaller top-k subset. These findings provide new insights for fine-grained classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。