通过注意力感知筛选减少计算量，让ViT模型提速2.8倍仍保持准确率
Speed-up of Vision Transformer Models by Attention-aware Token Filtering
- 在ViT编码器前加入注意力感知过滤模块，动态保留关键区域的视觉令牌
- 在不微调模型的前提下，使SigLIP模型推理速度提升2.8倍，召回率不变
- 适合需要高效部署ViT的图像检索、零样本分类等任务场景
视觉变换器（ViT）模型在图像嵌入提取方面取得突破，在零样本图像分类等任务中表现优异。然而，其计算开销较大。本文提出一种新型加速方法——注意力感知令牌过滤（ATF）。ATF包含两个核心：一个新设计的令牌过滤模块和一种过滤策略。该模块位于分块器与变换器编码器之间，不修改或微调编码器。它会过滤输入编码器的令牌，动态保留特定目标类型所在区域的令牌，并始终保留变换器编码器中静态高注意力区域的令牌。该策略在保持任务准确率的同时，显著减少输入编码器的令牌数量。在图像检索任务上的评估表明，ATF可使ViT模型SigLIP实现2.8倍的加速，同时维持原有的检索召回率。
原文摘要 · Abstract (English)
Vision Transformer (ViT) models have made breakthroughs in image embedding extraction, which provide state-of-the-art performance in tasks such as zero-shot image classification. However, the models suffer from a high computational burden. In this paper, we propose a novel speed-up method for ViT models called Attention-aware Token Filtering (ATF). ATF consists of two main ideas: a novel token filtering module and a filtering strategy. The token filtering module is introduced between a tokenizer and a transformer encoder of the ViT model, without modifying or fine-tuning of the transformer encoder. The module filters out tokens inputted to the encoder so that it keeps tokens in regions of specific object types dynamically and keeps tokens in regions that statically receive high attention in the transformer encoder. This filtering strategy maintains task accuracy while filtering out tokens inputted to the transformer encoder. Evaluation results on retrieval tasks show that ATF provides $2.8\times$ speed-up to a ViT model, SigLIP, while maintaining the retrieval recall rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。