修复视觉语言模型在少样本跨域场景下的微调失效问题
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
- 提出语义探针框架，通过修正视觉分类标记注意力来增强模态对齐
- 在4个基准上达到当前最优性能，适配各类微调方法
- 揭示文本结束符令牌更关注视觉样本，为跨域学习提供新视角
跨域少样本学习（CDFSL）旨在用少量样本将大规模预训练模型适配到特定目标域，但视觉语言模型（如CLIP）的少样本微调仍缺乏深入研究。我们建立多个基于CLIP的微调基线，发现适配器方法（如LoRA）在跨域场景中持续优于提示方法（如MaPLe），与域内情况相反。分析表明，LoRA的优势源于修正视觉分类标记的注意力坍缩，通过聚焦与文本相关的视觉区域提升模态对齐和类别分离。进一步发现，文本的结束符（EOS）标记对视觉样本具有更强注意力，而CLIP的标准对比损失对模态对齐约束较弱。基于此，我们提出可即插即用的语义探针（Semantic Probe）框架，适用于适配器和提示类方法。在四个CDFSL基准上的大量实验验证了该方案的有效性，实现当前最优性能，并同时惠及两类微调范式。代码将公开。
原文摘要 · Abstract (English)
Cross-Domain Few-Shot Learning (CDFSL) aims to adapt large-scale pretrained models to specialized target domains with limited samples, yet the few-shot fine-tuning of vision-language models like CLIP remains underexplored. By establishing multiple fine-tuning baselines of CLIP for CDFSL, we find adapter-based methods (e.g., LoRA) consistently outperform prompt-based ones (e.g., MaPLe), contrary to in-domain scenarios. To make those effective in-domain methods competitive again in CDFSL, we analyze this phenomenon and discover LoRA's superiority stems from rectifying the collapsed attention of visual CLS token, enhancing modality alignment and class separation by focusing on text-related visual regions. Further, we find textual EOS token exhibit much better attention to visual samples, and CLIP's standard contrastive loss weakly constrains modality alignment. Based on these insights, we propose Semantic Probe, a plug-and-play attention rectification framework for both adapter- and prompt-based methods. Extensive experiments on four CDFSL benchmarks validate our rationale, achieving state-of-the-art performance and benefiting both fine-tuning paradigms. Codes will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。