用图像本身选属性，比用类别名生成描述更准确可靠。
Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models
- 从图像中挑选视觉相关属性，而非依赖语言模型生成描述
- 新方法在ImageNet上达23.8%准确率，提升近8个百分点
- 无需训练，一分钟内完成，适合快速分析数据分布变化
一种常见的可解释零样本分类方法是让大语言模型（LLM）描述类别名称，再用生成的描述词提示CLIP模型。我们发现这些描述词缺乏自身视觉证据：去掉类别名后，ImageNet准确率从59.5%骤降至15.5%。原因在于描述词受标签条件影响，而非图像内容，导致在数据分布变化时严重误导。例如，LLM坚持草莓是红色，但ImageNet-Sketch中的草莓仅为无色线稿。因此，我们改从目标图像集合中筛选属性：在CLIP联合嵌入空间中对大量候选属性评分，保留每类得分最高的属性。以这种方式选取的无类别名属性提示，在ImageNet上达到23.8%准确率（相比LLM描述的15.5%），且在四个分布偏移的ImageNet变体上仍保持优势。重新从LLM池中筛选属性，证实选择机制才是关键。仅需每类一张图像，该方法比提示调优方法CoOp高3个百分点，耗时不足一分钟，无需学习软提示，决策透明。所选属性集由数据决定，可作为数据集的可读摘要，用于文字描述分布偏移。代码与结果见项目页面：https://ggare-cmu.github.io/AttributeSelect/
原文摘要 · Abstract (English)
A popular route to interpretable zero-shot classification asks a large language model (LLM) to describe each class name and prompts CLIP with the resulting descriptors. We show that these descriptors carry little visual evidence of their own: removing the class name from the prompt collapses ImageNet accuracy from 59.5% to 15.5%. The diagnosis is that the descriptors are conditioned on the label rather than on the images, so they describe the concept in general and mislead exactly when the data shifts; an LLM insists that strawberries are red, but every strawberry in ImageNet-Sketch is a colorless line drawing. We therefore select attributes from the target image collection instead: we score a large attribute pool against the images in CLIP's joint embedding space and keep the top-scoring attributes per class. Selected this way, class-name-free attribute prompts reach 23.8% on ImageNet (against 15.5% for LLM descriptors), the gain holds on four shifted ImageNet variants, and reselecting from the LLM's own pool isolates the selection mechanism as the cause. With one image per class, the selected attributes outperform the prompt-tuning method CoOp by 3 points while fitting in under a minute instead of 14 hours, with no learned soft prompt to obscure the decision. Because the attribute set is chosen by the data, it doubles as a readable summary of a dataset, which we use to describe distribution shift in words. Our code and results are available on our project page: https://ggare-cmu.github.io/AttributeSelect/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。