SAC-ViT通过分阶段剪枝与聚类，让ViT在低资源设备上更快更省力。
SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exit
- 先快速判断是否可提前退出，否则对关键区域做语义聚类
- 减少62%计算量，推理速度提升1.98倍，精度不变
- 适合部署在手机、嵌入式等算力受限的设备
视觉变换器（ViT）在全局建模上表现优异，但其注意力机制带来的二次计算复杂度限制了在资源受限设备上的部署。为此，我们提出语义感知聚类视觉变换器（SAC-ViT），一种非迭代的提升效率方法。SAC-ViT分为两个阶段：早期退出（EE）和语义感知聚类（SAC）。在EE阶段，下采样输入图像被处理以提取全局语义信息并生成初步预测结果；若未满足退出条件，则将信息聚类为目标与非目标标记。在SAC阶段，目标标记被映射回原图、裁剪并嵌入，再与来自EE阶段复用的非目标标记结合，在每个聚类内应用注意力机制。该两阶段设计经端到端优化，减少空间冗余，显著提升整体性能。大量实验表明，SAC-ViT将DeiT的浮点运算量减少62%，吞吐量提升1.98倍，且不损失精度。
原文摘要 · Abstract (English)
The Vision Transformer (ViT) excels in global modeling but faces deployment challenges on resource-constrained devices due to the quadratic computational complexity of its attention mechanism. To address this, we propose the Semantic-Aware Clustering Vision Transformer (SAC-ViT), a non-iterative approach to enhance ViT's computational efficiency. SAC-ViT operates in two stages: Early Exit (EE) and Semantic-Aware Clustering (SAC). In the EE stage, downsampled input images are processed to extract global semantic information and generate initial inference results. If these results do not meet the EE termination criteria, the information is clustered into target and non-target tokens. In the SAC stage, target tokens are mapped back to the original image, cropped, and embedded. These target tokens are then combined with reused non-target tokens from the EE stage, and the attention mechanism is applied within each cluster. This two-stage design, with end-to-end optimization, reduces spatial redundancy and enhances computational efficiency, significantly boosting overall ViT performance. Extensive experiments demonstrate the efficacy of SAC-ViT, reducing 62% of the FLOPs of DeiT and achieving 1.98 times throughput without compromising performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。