用大模型提升无源域适应性能,无需源数据即可应对标签分布变化。
LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation

- 利用视觉语言模型计算样本与文本标签相似度,识别未知类别。
- 通过相似度方差判断标签偏移类型,用高斯混合模型检测未知样本。
- 融合源模型与大模型知识,生成更准伪标签,适合跨域迁移任务。
无源通用域适应(SF-UniDA)在无源数据情况下,将预训练源模型适配至未标注的目标域,同时应对协变量和标签分布的变化。现有方法依赖低效的阈值调整与聚类技术。本文提出基于基础模型的框架(LFM),利用视觉-语言模型(VLM)计算目标样本与文本标签(包括由大语言模型提示生成的未知类别)的相似度。通过分析相似度得分的变异系数,确定标签偏移类型;采用二元高斯混合模型,基于另一相似度指标识别未知样本。在共识策略下,由初始化于源模型的目标模型对VLM生成的伪标签进行优化,融合源域与基础模型的知识。最终使用优化后的伪标签训练目标模型。在多种基准和所有可能标签偏移设置下的大量实验表明,该框架有效且显著优于现有方法。代码已开源。
原文摘要 · Abstract (English)
Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data. However, existing SF-UniDA methods rely on inefficient techniques such as threshold tuning and clustering. Foundation models (FMs), known for their generalization and zero-shot capabilities, remain underexplored in SF-UniDA. In this paper, we propose a framework that leverages foundation models (LFM) for SF-UniDA. We use a vision-language model (VLM) to compute similarities between target samples and text labels, including those for unknown classes generated by prompting a large language model. The label shift type is determined by analyzing the coefficient of variation of a similarity-based sample-level score. Unknown samples are identified using a binary Gaussian mixture model fitted to another similarity-based metric. Under a consensus strategy, the pseudo-labels generated by the VLM are refined by the target model initialized with the pre-trained source model, integrating knowledge from both the source domain and foundation models. Finally, these refined pseudo-labels are used to train the target model. Extensive experiments across all possible label shifts and multiple benchmarks demonstrate the effectiveness and superiority of our proposed LFM framework. Our code is available at https://github.com/iamjingli/LFM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。