用随机多提示微调,让CLIP在极端领域偏移下仍能少样本精准适配。
Multiple Stochastic Prompt Tuning for Few-shot Adaptation under Extreme Domain Shift
- 为每类设计多个可学习的随机提示,捕捉视觉分布偏移带来的多样性。
- 在极端领域偏移下,少样本测试准确率提升12.7个百分点。
- 适合所有类别同时部署的真实场景,无需分批处理。
基础视觉-语言模型(如CLIP)得益于大规模跨模态预训练,在多样图像-文本对上展现出强泛化能力。然而,当应用于视觉外观或语义类别存在显著分布偏移的目标数据集时,其性能常严重下降。现有少样本学习方法虽通过适配器或提示调优使CLIP适应下游任务,但未专门针对极端领域偏移进行设计。另一方面,部分跨域少样本学习工作虽考虑了领域偏移问题,却仅在每轮仅包含少数类别的分段式(episodic)设置下运行,限制了其在真实场景中的应用——真实场景需同时处理全部类别。为填补此空白,本文提出新型框架MIST(Multiple Stochastic Prompt Tuning),仅使用少量标注样本,即可高效适应具有极端分布偏移的数据集,且支持所有类别一次性处理。具体而言,我们为每类引入多个可学习提示,以有效捕捉由分布偏移引发的视觉表征多样性;进一步地,将这些提示建模为可学习的高斯分布,实现对提示参数空间的高效探索,降低因监督信号有限导致的过拟合。大量实验与最先进方法对比验证了该框架的有效性。
原文摘要 · Abstract (English)
Foundation Vision-Language Models (VLMs) like CLIP exhibit strong generalization capabilities due to large-scale pretraining on diverse image-text pairs. However, their performance often degrades when applied to target datasets with significant distribution shifts in both visual appearance and class semantics. Recent few-shot learning approaches adapt CLIP to downstream tasks using limited labeled data via adapter or prompt tuning, but are not specifically designed to handle such extreme domain shifts. Conversely, some works addressing cross-domain few-shot learning consider such domain-shifted scenarios but operate in an episodic setting with only a few classes per episode, limiting their applicability to real-world deployment, where all classes must be handled simultaneously. To address this gap, we propose a novel framework, MIST (Multiple Stochastic Prompt Tuning), for efficiently adapting CLIP to datasets with extreme distribution shifts using only a few labeled examples, in scenarios involving all classes at once. Specifically, we introduce multiple learnable prompts per class to effectively capture diverse modes in visual representations arising from distribution shifts. To further enhance generalization, these prompts are modeled as learnable Gaussian distributions, enabling efficient exploration of the prompt parameter space and reducing overfitting caused by limited supervision. Extensive experiments and comparisons with state-of-the-art methods demonstrate the effectiveness of the proposed framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。