arXiv:2412.00767cs.CVcs.CL2024-12被引 12

用文本提示增强少样本跨域迁移的多样性,不依赖源数据也能达到顶尖效果。

Prompt as Free Lunch: Enhancing Diversity in Source-Free Cross-domain Few-shot Learning through Semantic-Guided Prompting

  • 通过语义引导的提示词增加支持样本多样性,提升特征表达能力。
  • 在多个基准上性能媲美依赖源数据的先进方法,源无关设置下表现最佳。
  • 适合需要高效迁移且无源数据可用的研究场景,尤其关注泛化与多样性。

源无关跨域少样本学习(CD-FSL)旨在仅使用少量目标域样本,将预训练模型迁移到新领域,无需源域数据。该任务要求模型具备强泛化能力与特征表示能力,契合大模型特性。然而,大模型在跨域场景中因样本多样性不足易丧失表征能力。本文利用语义模态丰富的多样性,提出基于文本模态的增强策略。我们设计了SeGD-VPT框架,分两阶段:第一阶段通过为每个支持样本添加多样性提示词,生成不同输入以提升样本多样性;结合类别语义描述引导提示词学习,采用随机组合与选择文本以增强文本多样性;引入深度提示微调提升迁移能力。第一阶段结束后,带多样提示的支持样本输入CLIP主干网络生成增强特征。第二阶段基于生成特征训练分类器。大量实验验证,本方法在多个基准上性能媲美现有依赖源数据的最优模型,在源无关设定下表现最佳。

原文摘要 · Abstract (English)

The source-free cross-domain few-shot learning (CD-FSL) task aims to transfer pretrained models to target domains utilizing minimal samples, eliminating the need for source domain data. Addressing this issue requires models to have robust generalization abilities and strong feature representation, aligning with the characteristics of large-scale pretrained models. However, large-scale models tend to lose representational ability in cross-domain scenarios due to limited sample diversity. \zlh{Given the abundant diversity provided by semantic modality, this paper leverages textual modality to enhance training sample diversity with CLP model}, meanwhile improving model transfer efficiency. Specifically, we propose the SeGD-VPT framework, which is divided into two phases. The first step aims to increase feature diversity by adding diversity prompts to each support sample, thereby generating varying input and enhancing sample diversity. Furthermore, we use diversity descriptions of classes to guide semantically meaningful learning of diversity prompts, proposing random combinations and selections of texts to increase textual diversity. Additionally, deep prompt tuning is introduced to enhance the model's transfer capability. After training of the first step, support samples with different diversity prompts are input into the CLIP backbone to generate enhanced features. After generation, the second phase trains classifiers using the generated features. Extensive experimental results across several benchmarks verify our method is comparable to SOTA source-utilized models and attain the best performance under the source-free CD-FSL setting.

少样本学习跨域迁移提示工程CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。