arXiv:2510.02270cs.CVcs.AI2025-10ACL被引 2

通过细粒度特征融合,让CLIP模型在无监督下更精准识别图像细微差异。

microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification

  • 用注意力机制从局部图像块提取细粒度特征,与全局特征融合对齐。
  • 在13个细粒度分类数据集上平均提升2.90%准确率,仅需轻量级调整。
  • 适合希望不依赖标注数据提升图像分类精度的研究者或工程师。

基于视觉语言模型(VLM)的CLIP在零样本迁移中表现强劲,但在细粒度图像分类任务中受限于对微观局部线索的敏感性。现有方法通过将大语言模型(LLM)描述与CLIP的[CLS]标记对齐来注入细粒度知识,但忽略了空间精度。本文提出microCLIP,一种自训练框架,通过细粒度线索联合优化CLIP的视觉与文本表征。核心是轻量级TokenFusion模块中的显著性导向注意力池化(SOAP),从图像块嵌入构建显著性引导的[FG]标记,并与全局[CLS]标记融合实现粗-细对齐。为稳定适应过程,引入双头LLM生成分类器:一个冻结分类器通过多视角对齐提供稳定的伪标签先验;另一个可学习分类器初始来自LLM描述并经由TokenFusion微调。此外,动态知识聚合机制以凸组合方式融合固定LLM/CLIP先验与TokenFusion的演进逻辑,迭代优化伪标签。该框架有效挖掘了CLIP中的潜在细粒度信号,在13个细粒度基准上实现平均2.90%的准确率提升,且仅需轻量级适应。代码已开源。

原文摘要 · Abstract (English)

Unsupervised adaptation of CLIP-based vision-language models (VLMs) for fine-grained image classification requires sensitivity to microscopic local cues. While CLIP exhibits strong zero-shot transfer, its reliance on coarse global features restricts its performance on fine-grained classification tasks. Prior efforts inject fine-grained knowledge by aligning large language model (LLM) descriptions with the CLIP $\texttt{[CLS]}$ token; however, this approach overlooks spatial precision. We propose $\textbf{microCLIP}$, a self-training framework that jointly refines CLIP's visual and textual representations using fine-grained cues. At its core is Saliency-Oriented Attention Pooling (SOAP) within a lightweight TokenFusion module, which builds a saliency-guided $\texttt{[FG]}$ token from patch embeddings and fuses it with the global $\texttt{[CLS]}$ token for coarse-fine alignment. To stabilize adaptation, we introduce a two-headed LLM-derived classifier: a frozen classifier that, via multi-view alignment, provides a stable text-based prior for pseudo-labeling, and a learnable classifier initialized from LLM descriptions and fine-tuned with TokenFusion. We further develop Dynamic Knowledge Aggregation, which convexly combines fixed LLM/CLIP priors with TokenFusion's evolving logits to iteratively refine pseudo-labels. Together, these components uncover latent fine-grained signals in CLIP, yielding a consistent $2.90\%$ average accuracy gain across 13 fine-grained benchmarks while requiring only light adaptation. Our code is available at https://github.com/sathiiii/microCLIP.

CLIP细粒度分类自训练视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。