通过提示调优与头选择，让大视觉语言模型在少样本分类上表现超越CLIP
Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection
- 用提示条件化提升视觉特征区分度，挖掘模型内部注意力头潜力
- 无需训练的头集成分类器在12个数据集上达到先进水平
- 适合研究少样本学习、模型可解释性及视觉语言融合的学者
当前大视觉语言模型（LVLMs）在图像描述、视觉问答和OCR等零样本任务中表现优异，但在图像分类任务上性能不佳，低于基于CLIP的方法。尽管许多LVLMs采用预训练于CLIP的视觉编码器，其性能瓶颈并非源于架构限制。在CLIP中，视觉与文本编码器的分离导致分类偏向名称匹配而非联合视觉-文本推理。本文表明，尽管原始性能较差，LVLMs可通过提示条件化提升图像特征的类别可分性，且其内部表示（特别是注意力头）在零样本和少样本分类中优于模型整体。我们提出无训练的头集成分类器（HEC），受高斯判别分析启发，筛选最具判别力的视觉与文本注意力头并组合成分类器。实验证明，HEC在12个数据集上的少样本与零样本分类任务中均达到最优性能。
原文摘要 · Abstract (English)
Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。