无需人工标注，让视觉语言模型自动高效适配下游任务。
Fine-tuning Pre-trained Vision-Language Models in a Human-Annotation-Free Manner
- 双模型协作机制，用正负文本提示动态判断伪标签可靠性。
- 在多个数据集上超越现有无监督方法，接近有监督少样本表现。
- 适合缺乏标注数据但需快速部署视觉语言模型的场景。
大规模视觉语言模型（如CLIP）具备强大的零样本泛化能力，但将其适配到下游任务通常需要昂贵的人工标注数据。现有无监督自训练方法依赖伪标签，但常因置信度筛选不可靠、确认偏误及低置信度样本利用不足而受限。本文提出协同微调（CoFT），一种基于双模型跨模态协作的无监督适配框架。CoFT采用双提示学习策略，通过正负文本提示显式建模样本级伪标签纯净度，无需手工设定阈值或噪声假设。负提示还对轻量级视觉适配模块进行正则化，增强在噪声监督下的鲁棒性。CoFT采用两阶段训练：先在高置信度样本上进行参数高效微调，再由协同过滤的伪标签引导全模型微调。在此基础上，CoFT+进一步通过迭代微调、动量对比学习和大语言模型生成提示提升性能。大量实验表明，CoFT在多个基准上持续优于现有无监督方法，甚至超越部分少样本有监督基线。
原文摘要 · Abstract (English)
Large-scale vision-language models (VLMs) such as CLIP exhibit strong zero-shot generalization, but adapting them to downstream tasks typically requires costly labeled data. Existing unsupervised self-training methods rely on pseudo-labeling, yet often suffer from unreliable confidence filtering, confirmation bias, and underutilization of low-confidence samples. We propose Collaborative Fine-Tuning (CoFT), an unsupervised adaptation framework that leverages unlabeled data through a dual-model, cross-modal collaboration mechanism. CoFT introduces a dual-prompt learning strategy with positive and negative textual prompts to explicitly model pseudo-label cleanliness in a sample-dependent manner, removing the need for hand-crafted thresholds or noise assumptions. The negative prompt also regularizes lightweight visual adaptation modules, improving robustness under noisy supervision. CoFT employs a two-phase training scheme, transitioning from parameter-efficient fine-tuning on high-confidence samples to full fine-tuning guided by collaboratively filtered pseudo-labels. Building on CoFT, CoFT+ further enhances adaptation via iterative fine-tuning, momentum contrastive learning, and LLM-generated prompts. Extensive experiments demonstrate consistent gains over existing unsupervised methods and even few-shot supervised baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。