用超网络让小模型实现大模型的零样本图像分类能力
HyperCLIP: Adapting Vision-Language models with Hypernetworks
- 用超网络动态生成图像编码器权重，适配不同文本输入
- 在ImageNet上提升零样本准确率3%，CIFAR-100上提升5%
- 适合资源受限场景下的快速部署与零样本任务适配
基于对比学习的自监督视觉-语言模型是当前人工智能视觉任务的主流方法。这类模型的成功源于大规模网络数据集的训练，但需要大型视觉组件以从广阔数据域中学习强大且通用的表征，这给部署带来挑战，尤其在资源受限环境下。为此，我们提出一种新型视觉-语言架构HyperCLIP，采用小型图像编码器结合超网络，根据新文本输入动态调整图像编码器权重。模型的三个组件（超网络、图像编码器、文本编码器）均联合预训练并端到端优化。训练完成后，仅需一次前向传播即可为任意任务生成零样本部署友好的图像分类器。HyperCLIP使使用小图像编码器的SigLIP模型在ImageNet上的零样本准确率提升最高达3%，在CIFAR-100上提升5%，且训练开销极低。
原文摘要 · Abstract (English)
Self-supervised vision-language models trained with contrastive objectives form the basis of current state-of-the-art methods in AI vision tasks. The success of these models is a direct consequence of the huge web-scale datasets used to train them, but they require correspondingly large vision components to properly learn powerful and general representations from such a broad data domain. This poses a challenge for deploying large vision-language models, especially in resource-constrained environments. To address this, we propose an alternate vision-language architecture, called HyperCLIP, that uses a small image encoder along with a hypernetwork that dynamically adapts image encoder weights to each new set of text inputs. All three components of the model (hypernetwork, image encoder, and text encoder) are pre-trained jointly end-to-end, and with a trained HyperCLIP model, we can generate new zero-shot deployment-friendly image classifiers for any task with a single forward pass through the text encoder and hypernetwork. HyperCLIP increases the zero-shot accuracy of SigLIP trained models with small image encoders by up to 3% on ImageNet and 5% on CIFAR-100 with minimal training throughput overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。