分析提示学习如何改变视觉语言模型的语义概念表达
What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
- 用自然语言词典解析提示学习前后文本嵌入的语义构成
- 平均只有1.6个初始关键词在学习后仍保留在前10名
- 揭示了图像对齐概念方向更具损失敏感性的几何原因
提示学习通过优化连续提示向量来适应视觉语言模型（如CLIP），但学习得到的提示难以用自然语言解释。我们提出PromptSpLiCE，一种后处理方法，将每个类别条件文本嵌入表示为固定自然语言词典中概念的稀疏组合。利用学习前后相同的词典，可对比概念分布的变化。我们在CoOp方法上评估了该方法，覆盖11个图像分类数据集。结果表明，概念分布发生显著变化：平均仅有1.6个初始前10概念在学习后仍位于前10。跨数据集分析显示，概念分布变化程度与准确率提升呈正相关。此外，我们推导出局部梯度表达，提供了几何直观：与当前提示方向不同的图像对齐概念方向具有更高的损失敏感性。
原文摘要 · Abstract (English)
Prompt learning adapts vision-language models such as CLIP by optimizing continuous prompt vectors, but the learned prompts are difficult to interpret in natural language. We present PromptSpLiCE, a post-hoc method that expresses each class-conditioned text embedding as a sparse combination of concepts from a fixed natural-language dictionary. Using the same dictionary before and after prompt learning allows us to compare changes in their concept profiles. We evaluate PromptSpLiCE on CoOp, a representative prompt-learning method, across 11 image-classification datasets. The concept profiles change substantially: on average, only 1.6 of the initial top-10 concepts remain in the top 10 after learning. Across datasets, profile change is positively associated with accuracy gain. We also derive a local gradient expression that provides geometric intuition for why image-aligned concept directions distinct from the current prompt can have greater loss sensitivity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。