用大模型注意力头提取特征，少样本下图像语言分类效果领先
Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features
- 不微调大模型，仅用其注意力头激活值提取特征
- 少样本下在多个视觉语言分类任务上达到顶尖性能
- 方法简单高效，适合资源有限的少样本场景
生成式大型多模态模型（如 LLaVA 和 Qwen-VL）在多种视觉-语言任务中表现优异。然而，其生成输出并不专精于图像分类、多选题 VQA 等需要离散标签的视觉-语言分类任务。关键挑战在于如何从生成式多模态模型中提取有效特征。为此，我们提出一种无需微调的方法——稀疏注意力向量（SAVs），利用模型中激活的稀疏注意力头（少于5%的头）作为强特征表示。仅需少量样本，SAVs 在多个视觉-语言分类任务上超越多种少样本及微调基线方法。实验表明，SAVs 性能可随样本增加而提升，并能泛化至相似任务，证明其兼具有效性与鲁棒性。
原文摘要 · Abstract (English)
Generative Large Multimodal Models (LMMs) like LLaVA and Qwen-VL excel at a wide variety of vision-language (VL) tasks. Despite strong performance, LMMs' generative outputs are not specialized for vision-language classification tasks (i.e., tasks with vision-language inputs and discrete labels) such as image classification and multiple-choice VQA. One key challenge in utilizing LMMs for these tasks is the extraction of useful features from generative LMMs. To overcome this, we propose an approach that leverages multimodal feature extraction from the LMM's latent space. Toward this end, we present Sparse Attention Vectors (SAVs) -- a finetuning-free method that leverages sparse attention head activations (fewer than 5% of the heads) in LMMs as strong feature representations. With only few-shot examples, SAVs demonstrate state-of-the-art performance compared to a variety of few-shot and finetuned baselines on a collection of vision-language classification tasks. Our experiments also imply that SAVs can scale in performance with additional examples and generalize to similar tasks, establishing SAVs as both effective and robust multimodal feature representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。