提出CoPA方法,让原型和图像用不同变换,提升跨域少样本分类性能。
Mind the Gap Between Prototypes and Images in Cross-domain Finetuning
- 为原型和图像分别设计独立变换,避免共享变换带来的限制
- 在Meta-Dataset上达到新最佳,验证损失最小且特征聚类更优
- 适合关注跨域少样本学习与表示分离的科研人员
在跨域少样本分类(CFC)中,现有方法通常在冻结预训练主干网络的基础上,通过简单变换头将嵌入投影到任务特定度量空间,通过计算图像实例与原型表示之间的相似性进行分类。该框架隐含假设:原型与图像嵌入采用相同的表示变换。然而本文发现,从冻结主干中提取的原型与图像嵌入之间存在自然差距,类似模态差距;而统一应用相同变换会限制最优表示探索,并压缩两者间的差距。为此,提出对比原型-图像适配方法(CoPA),借鉴CLIP思想,将原型视为文本提示,分别为原型和图像设计独立变换。在Meta-Dataset上的大量实验表明,CoPA以更高效的方式实现当前最优性能。进一步分析显示,CoPA能学习更优的表示聚类,扩大原型与图像间差距,并在扩大后的差距下取得最小验证损失。
原文摘要 · Abstract (English)
In cross-domain few-shot classification (CFC), recent works mainly focus on adapting a simple transformation head on top of a frozen pre-trained backbone with few labeled data to project embeddings into a task-specific metric space where classification can be performed by measuring similarities between image instance and prototype representations. Technically, an assumption implicitly adopted in such a framework is that the prototype and image instance embeddings share the same representation transformation. However, in this paper, we find that there naturally exists a gap, which resembles the modality gap, between the prototype and image instance embeddings extracted from the frozen pre-trained backbone, and simply applying the same transformation during the adaptation phase constrains exploring the optimal representations and shrinks the gap between prototype and image representations. To solve this problem, we propose a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformations respectively for prototypes and images similarly to CLIP by treating prototypes as text prompts. Extensive experiments on Meta-Dataset demonstrate that CoPA achieves the state-of-the-art performance more efficiently. Meanwhile, further analyses also indicate that CoPA can learn better representation clusters, enlarge the gap, and achieve minimal validation loss at the enlarged gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。