用概念原型约束提示词，让CLIP少跑偏、泛化更好。
Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation
- 用冻结的概念原型约束可学习的类别提示词
- 在DTD和EuroSAT上比CoOp提升2.9和0.6分
- 适合细粒度分类且对概念语义要求高场景
少样本提示学习是适配CLIP到下游任务的有效方法，但仅优化类别提示容易过拟合基础类监督，削弱对未见类的迁移能力。我们提出概念约束提示学习（CCPL），一种轻量级正则化框架，将可学习的类别提示锚定在冻结的概念级文本原型上，不更新CLIP编码器。CCPL学习一组共享上下文标记，通过追加类别名称生成类别提示，并从类别级概念库构建冻结的概念原型。训练时，文本空间余弦一致性目标使可学习的类别提示嵌入与冻结的概念原型对齐；概念丢弃提供额外正则化，防止过度依赖固定概念列表。推理时，可通过可控集成权重alpha选择性融合类别提示和概念原型的得分。默认配置使用文本空间概念正则化λ=0.5，概念丢弃概率p=0.3，弱概念引导融合（alpha=0.1），无基于KL散度的预测一致性项。在相同自动生成的回退划分下，实验表明，相比CoOp，CCPL在DTD上提升基类到新类调和均值0.6，在EuroSAT上提升2.9，而在OxfordPets上变化为-0.1。消融实验表明文本空间概念正则化始终有益，而最佳概念引导推理强度受数据集和协议影响。结果表明，当概念原型与数据集语义自然对齐时，概念约束最有效，并指出细粒度类别是当前边界条件。代码已发布于：https://github.com/richael-sang/concept-constrained-prompt-learning。
原文摘要 · Abstract (English)
Few-shot prompt learning is an effective strategy for adapting CLIP to downstream tasks, but class-only prompt optimization can overfit base-class supervision and weaken transfer to unseen classes. We propose Concept-Constrained Prompt Learning (CCPL), a lightweight regularization framework that anchors learnable class prompts to frozen concept-level text prototypes without updating CLIP encoders. CCPL learns a set of shared context tokens, instantiates class prompts by appending class names, and constructs frozen concept prototypes from a class-level concept bank. During training, a text-space cosine consistency objective aligns learnable class-prompt embeddings with frozen concept prototypes; concept dropout provides additional regularization against over-reliance on fixed concept lists. At inference, CCPL optionally fuses class-prompt logits with concept-prototype logits using a controllable ensemble weight alpha. Our default configuration uses text-space concept regularization lambda = 0.5, concept dropout p = 0.3 and weak concept-guided fusion (alpha = 0.1), with no KL-based prediction consistency term. Experiments under identical automatically-generated fallback splits show that CCPL improves the base-to-new harmonic mean on DTD (+0.6) and EuroSAT (+2.9) compared with CoOp, while remaining near-neutral on OxfordPets (-0.1). Ablations indicate that text-space concept regularization is consistently beneficial, while the best concept-guided inference strength is dataset- and protocol-sensitive. These results suggest concept constraints are most effective when concept prototypes align naturally with dataset semantics, and identify fine-grained categories as a current boundary condition. The code is released at: https://github.com/richael-sang/concept-constrained-prompt-learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。