通过分离剪枝与通道选择，高效压缩ViT模型且保持高精度
ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT
- 分模块处理：自注意力层用结构化剪枝，前馈层用无训练通道筛选
- 在ViT-MAE-Huge上减少39.4%计算量，精度提升1.64个百分点
- 无需重训练，适配多种视觉任务，部署效率显著提升
视觉变换器（ViTs）在多个视觉任务中表现卓越，但其部署常受限于高昂的计算成本。现有结构化剪枝和令牌压缩方法分别面临冗长重训练和层间依赖问题。本文提出ToaSt，一种解耦框架，针对ViT不同组件采用专用策略：对多头自注意力模块实施耦合头级结构化剪枝，利用注意力特性增强鲁棒性；对占总计算量超60%的前馈网络，提出无需训练的令牌通道选择（TCS）方法，在推理时过滤冗余噪声通道。在九种不同模型（包括DeiT、ViT-MAE、Swin Transformer）上广泛评估显示，ToaSt在准确率与效率之间取得更优平衡，持续优于现有基线。在ViT-MAE-Huge上实现88.52%准确率（+1.64%p），FLOPs降低39.4%。该方法还能有效迁移至多种下游任务（如COCO检测、ADE20K分割、CIFAR-100分类），在COCO上获得52.2 vs 51.9 mAP。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have achieved remarkable success across various vision tasks, yet their deployment is often hindered by prohibitive computational costs. While structured weight pruning and token compression have emerged as promising solutions, they suffer from prolonged retraining and inter-layer dependencies that complicate optimization, respectively. We propose ToaSt, a decoupled framework applying specialized strategies to distinct ViT components. We apply coupled head-wise structured pruning to Multi-Head Self-Attention modules, leveraging attention operation characteristics to enhance robustness. For Feed-Forward Networks (over 60% of FLOPs), we introduce Token Channel Selection (TCS), a training-free method that filters redundant noise channels at inference time. Extensive evaluations across nine diverse models, including DeiT, ViT-MAE, and Swin Transformer, demonstrate that ToaSt achieves superior trade-offs between accuracy and efficiency, consistently outperforming existing baselines. On ViT-MAE-Huge, ToaSt achieves 88.52% accuracy (+1.64%p) with 39.4% FLOPs reduction. ToaSt also transfers effectively to diverse downstream tasks (COCO detection, ADE20K segmentation, CIFAR-100 classification), achieving 52.2 versus 51.9 mAP on COCO. Code: github.com/SHANNonLab-HUFS/ToaSt
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。