用注意力选择关键视觉区域,提升模型对全局语义的理解能力。
From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection

- 通过注意力机制在图像和特征空间中引导裁剪,聚焦重要区域。
- 在零样本分类任务上超越现有方法,达到当前最优性能。
- 无需训练即可使用,适合快速部署于新场景。
预训练视觉语言模型(如CLIP)在下游任务中展现出出色的零样本能力。以往研究强调随机裁剪等视觉增强技术在与大语言模型生成的细粒度类别描述对齐中的关键作用,能通过多视角信息显著提升零样本性能。然而,这些增强技术的固有随机性会引入背景干扰,导致模型过度关注局部细节,削弱全局语义理解。为此,我们提出一种从局部细节到全局上下文的注意力选择方法(ABS),在原始图像和特征空间中均采用注意力引导裁剪,并通过策略性特征选择补充全局语义信息。此外,引入软匹配技术有效筛选LLM生成的描述以实现更好对齐。ABS在分布外泛化和零样本分类任务上达到当前最优表现,且为无训练方法,甚至可媲美少样本及测试时自适应方法。代码已开源。
原文摘要 · Abstract (English)
Pretrained vision-language models (VLMs), e.g., CLIP, demonstrate impressive zero-shot capabilities on downstream tasks. Prior research highlights the crucial role of visual augmentation techniques, like random cropping, in alignment with fine-grained class descriptions generated by large language models (LLMs), significantly enhancing zero-shot performance by incorporating multi-view information. However, the inherent randomness of these augmentations can inevitably introduce background artifacts and cause models to overly focus on local details, compromising global semantic understanding. To address these issues, we propose an \textbf{A}ttention-\textbf{B}ased \textbf{S}election (\textbf{ABS}) method from local details to global context, which applies attention-guided cropping in both raw images and feature space, supplement global semantic information through strategic feature selection. Additionally, we introduce a soft matching technique to effectively filter LLM descriptions for better alignment. \textbf{ABS} achieves state-of-the-art performance on out-of-distribution generalization and zero-shot classification tasks. Notably, \textbf{ABS} is training-free and even rivals few-shot and test-time adaptation methods. Our code is available at \href{https://github.com/BIT-DA/ABS}{\textcolor{darkgreen}{https://github.com/BIT-DA/ABS}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。