让视觉模型主动筛选关键图像块,提升识别精度
Screening Is Effective for Visual Recognition

- 引入筛选机制,独立评估每张图像块相关性
- 在分类任务中优于传统ViT,提升特征聚合效率
- 适合需要精准特征选择的视觉识别场景
视觉变换器(ViT)广泛用于建模图像块间的全局依赖关系。然而,其核心自注意力机制对所有图像块分配归一化的相对权重,难以独立评估块间相关性。图像中常含大量背景或冗余块,而自注意力无法显式排除无关块,可能引入无效信息。为解决此问题,我们提出VisionScreen,将语言建模中的筛选机制扩展至视觉识别。该方法将图像块视为二维网格上的令牌,基于查询-键相似性扩展绝对相关性评估至二维空间,使每个块仅聚合内容与空间上相关的块,无需依赖块间竞争。在图像分类基准测试中,该方法表现优于传统ViT,表明筛选机制在视觉识别中有效,可作为软注意力的替代方案。
原文摘要 · Abstract (English)
Vision Transformer (ViT) has been widely used as a powerful framework for modeling global dependencies among image patches. However, its core component, self-attention assigns softmax-normalized relative weights to all patches, making it difficult to evaluate the relevance between patches independently. In visual recognition, images often contain many background or redundant patches, yet self-attention cannot explicitly reject such irrelevant patches, which may introduce unnecessary information into feature aggregation. To address this limitation, Screening has been proposed in the field of language modeling, where the relevance of each token is independently evaluated based on query-key similarity and low-relevance tokens are explicitly excluded through thresholding. In this work, we propose VisionScreen, a new vision model that extends Screening mechanism to visual recognition. VisionScreen treats image patches as tokens arranged on a two-dimensional grid and extends absolute relevance estimation based on query-key similarity to the two-dimensional spatial domain. This allows each patch to selectively aggregate only content-wise and spatially relevant patches without relying on competition among patches. Experiments on image classification benchmarks demonstrate that the proposed method outperforms conventional ViT. These results suggest that Screening can be effective for visual recognition, offering an alternative to relative feature aggregation based on softmax attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。