用预训练模型的分类头做语义原型，零样本对齐更高效
Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
- 把预训练视觉模型的分类头权重重用为语义原型
- 在跨模态检索、零/少样本分类上提升准确率
- 无需额外标注数据，适合资源有限场景
视觉语言模型（VLMs）通过将图像与文本映射到共享空间，在零样本分类和跨模态检索任务中表现优异，但需大规模成对数据进行端到端训练。现有后处理对齐方法虽降低计算成本，仍依赖大量成对数据。本文探索将预训练视觉模型的分类头权重重用为语义原型。这些通常被丢弃的权重，可作为零样本对齐的语义锚点，并通过与真实图像-文本对混合实现稳健的数据增强。实验表明，该方法与多种先进后处理对齐技术结合，均显著提升跨模态检索、零样本及少样本分类的性能。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) excel at tasks like zero-shot classification and cross-modal retrieval by mapping images and text to a shared space, but this requires expensive end-to-end training with massive paired datasets. Current post-hoc alignment methods reduce computational costs by connecting pretrained encoders through lightweight mappings, yet still demand substantial paired data. In this work, we investigate the potential of repurposing the classification heads of pretrained vision models as semantic prototypes. The recycling of these weights, typically discarded after pretraining, unlocks two distinct capabilities: it enables zero-shot alignment by using weights as semantic anchors, and serves as a robust data augmentation strategy by mixing these prototypes with real image-text pairs. We demonstrate that integrating our approach with several state-of-the-art post-hoc alignment techniques consistently boosts accuracy in cross-modal retrieval, zero- and few-shot classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。