提出块级对称剪枝融合方法，提升视觉Transformer效率与精度
Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers
- 基于块的对称剪枝与融合，联合优化查询与键令牌
- 在所有剪枝比例下超越现有方法，图像分类准确率提升1.3%-2.0%
- 适用于追求高效高精度视觉Transformer的工程部署场景
视觉Transformer（ViT）在各类视觉任务中表现优异，但其高昂的计算成本限制了实际应用。现有方法通过剪枝无关紧要的令牌来降低ViT的$O(n^2)$复杂度，但通常独立剪枝查询（Q）和键（K）令牌，忽视了令牌间交互，导致性能下降。为此，我们提出一种新的块级对称剪枝与融合方法（BSPF-ViT），联合优化Q/K令牌的剪枝。不同于仅考虑单一方向的方法，本方法评估每个令牌及其邻域，结合令牌交互决定保留哪些令牌，并通过相似性融合压缩保留令牌，保留关键信息的同时降低计算开销。共享的Q/K权重生成对称注意力矩阵，可仅剪枝上三角部分以加速。BSPF-ViT在所有剪枝水平上均优于当前最优的ViT方法，在DeiT-T上准确率提升1.3%，DeiT-S上提升2.0%，同时计算量减少50%。该方法在多种ViT架构上实现40%的速度提升且精度更高。
原文摘要 · Abstract (English)
Vision Transformer (ViT) has achieved impressive results across various vision tasks, yet its high computational cost limits practical applications. Recent methods have aimed to reduce ViT's $O(n^2)$ complexity by pruning unimportant tokens. However, these techniques often sacrifice accuracy by independently pruning query (Q) and key (K) tokens, leading to performance degradation due to overlooked token interactions. To address this limitation, we introduce a novel {\bf Block-based Symmetric Pruning and Fusion} for efficient ViT (BSPF-ViT) that optimizes the pruning of Q/K tokens jointly. Unlike previous methods that consider only a single direction, our approach evaluates each token and its neighbors to decide which tokens to retain by taking token interaction into account. The retained tokens are compressed through a similarity fusion step, preserving key information while reducing computational costs. The shared weights of Q/K tokens create a symmetric attention matrix, allowing pruning only the upper triangular part for speed up. BSPF-ViT consistently outperforms state-of-the-art ViT methods at all pruning levels, increasing ImageNet classification accuracy by 1.3% on DeiT-T and 2.0% on DeiT-S, while reducing computational overhead by 50%. It achieves 40% speedup with improved accuracy across various ViTs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。