arXiv:2603.05235cs.AI2026-03被引 6

通过唤醒被遗忘的文本层,提升少样本跨域学习性能

Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learning

  • 发现并重用文本编码器中被忽略的中间层信息
  • 在多个数据集上实现显著性能提升,最高达8.2%准确率增益
  • 适用于医疗、遥感等少样本跨域场景,尤其适合迁移学习研究者

源无关跨域少样本学习(SF-CDFSL)关注在目标域(如医学或卫星图像)中仅用少量数据进行微调的问题。尽管近期研究表明,CLIP 的文本编码器对跨域任务更具泛化性,但本文发现:移除文本编码器的部分中间层反而能提升 SF-CDFSL 性能,这些被移除的层被称为「丢失层」。我们深入探究该现象,发现这些层中的信息实际有益,但因视觉与文本之间的鸿沟,导致其未能被充分利用,从而显得冗余。基于此,我们提出一种新方法,旨在从层到编码器层级重新激活并利用这些被遗忘的信息,引导视觉分支在领域偏移下重新学习。实验在多种设置、主干网络(CLIP、SigLip、PE-Core)及任务(4 个 CDFSL 数据集和 10 个 Meta-dataset 数据集)上验证了方法的有效性,显著提升了模型性能。代码已开源:https://github.com/zhenyuZ-HUST/CVPR26-VtT。

原文摘要 · Abstract (English)

Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where CLIP has recently shown promising results due to its generalizability to downstream tasks. Current works indicate CLIP's text encoder is more suitable for cross-domain tasks, however, we find that \textbf{removing certain middle layers of the text encoder can effectively improve performance in SF-CDFSL}, which we call the Lost Layers. In this paper, we delve into this phenomenon for a deeper understanding. We discover that instead of being harmful for the SF-CDFSL task, the information in these layers is actually beneficial, but visual gaps prevent this useful information from being fully utilized, making these layers seem redundant. Based on this understanding, unlike current works that simply remove these layers, we propose a method to teachs the model to \textbf{re-utilize} information in these lost layers at both the layer and encoder levels, guiding the re-learning of the visual branch under domain shifts. Our approach effectively addresses the issue of underutilized information in the text encoder. Extensive experiments across various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 10 Meta-dataset datasets) demonstrate the effectiveness of our method. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-VtT.

少样本学习跨域迁移CLIP文本编码器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。