让CLIP模型轻松识别罕见类别，无需重新训练。
LiteEmbed: Adapting CLIP to Rare Classes
- 通过优化文本嵌入的子空间，实现轻量级个性化
- 在多个任务中显著提升罕见类别的识别效果
- 适合需要快速添加新类别的实际应用
大规模视觉-语言模型如CLIP在零样本识别上表现强劲，但在预训练中罕见的类别（包括新兴实体和文化特异性类别）上表现不佳。我们提出LiteEmbed，一种轻量级框架，可在不重训练编码器的情况下对CLIP进行少样本个性化。LiteEmbed在CLIP词表内执行基于主成分分析（PCA）分解的子空间引导文本嵌入优化，分离粗粒度语义方向与细粒度变化。通过粗对齐与细分离两个互补目标，同时保持全局语义一致性并增强视觉相似类别间的可区分性。优化后的嵌入可即插即用，在分类、检索、分割和检测任务中无缝替代CLIP原始文本特征。大量实验表明，相比现有方法有显著提升，验证了LiteEmbed在适应低频、罕见或未见类别上的有效性。
原文摘要 · Abstract (English)
Large-scale vision-language models such as CLIP achieve strong zero-shot recognition but struggle with classes that are rarely seen during pretraining, including newly emerging entities and culturally specific categories. We introduce LiteEmbed, a lightweight framework for few-shot personalization of CLIP that enables new classes to be added without retraining its encoders. LiteEmbed performs subspace-guided optimization of text embeddings within CLIP's vocabulary, leveraging a PCA-based decomposition that disentangles coarse semantic directions from fine-grained variations. Two complementary objectives, coarse alignment and fine separation, jointly preserve global semantic consistency while enhancing discriminability among visually similar classes. Once optimized, the embeddings are plug-and-play, seamlessly substituting CLIP's original text features across classification, retrieval, segmentation, and detection tasks. Extensive experiments demonstrate substantial gains over prior methods, establishing LiteEmbed as an effective approach for adapting CLIP to underrepresented, rare, or unseen classes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。