利用大模型协同迁移知识,提升少样本图像分类性能
Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning
- 用CLIP作教师,通过无监督任务蒸馏语义对齐的视觉知识
- 在四个数据集上超越现有方法,显著提升少样本分类准确率
- 无需训练,适合快速部署到各类少样本学习场景
少样本学习(FSL)旨在用极少训练样本分类新类别。现有方法虽借助小模型的语义知识缓解数据稀缺,但常引入噪声和偏差。本文提出协同知识迁移框架SynTrans,从大模型中有效转移多样且互补的知识,赋能现成的少样本学习器。SynTrans以CLIP为强教师,少样本视觉编码器为弱学生,通过无监督代理任务蒸馏语义对齐的视觉知识;随后,训练无关的协同知识挖掘模块促进大模型间协作,提取高质量语义知识;再通过视觉-语义桥接模块实现视觉与语义空间双向知识迁移,将显式视觉和隐式语义知识转化为类别特定分类器权重;最后,引入视觉权重生成器与语义权重重构器,自适应构建最优多模态FSL分类器。在四个FSL数据集上的实验表明,即使搭配简单少样本视觉编码器,SynTrans仍显著优于当前最先进方法。
原文摘要 · Abstract (English)
Few-shot learning (FSL) addresses the challenge of classifying novel classes with limited training samples. While some methods leverage semantic knowledge from smaller-scale models to mitigate data scarcity, these approaches often introduce noise and bias due to the data's inherent simplicity. In this paper, we propose a novel framework, Synergistic Knowledge Transfer (SynTrans), which effectively transfers diverse and complementary knowledge from large multimodal models to empower the off-the-shelf few-shot learner. Specifically, SynTrans employs CLIP as a robust teacher and uses a few-shot vision encoder as a weak student, distilling semantic-aligned visual knowledge via an unsupervised proxy task. Subsequently, a training-free synergistic knowledge mining module facilitates collaboration among large multimodal models to extract high-quality semantic knowledge. Building upon this, a visual-semantic bridging module enables bi-directional knowledge transfer between visual and semantic spaces, transforming explicit visual and implicit semantic knowledge into category-specific classifier weights. Finally, SynTrans introduces a visual weight generator and a semantic weight reconstructor to adaptively construct optimal multimodal FSL classifiers. Experimental results on four FSL datasets demonstrate that SynTrans, even when paired with a simple few-shot vision encoder, significantly outperforms current state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。