arXiv:2603.12222cs.CVcs.LG2026-03

提出一种端到端的多粒度自动剪枝框架,高效压缩视觉Transformer。

HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers

  • 通过引入随机门控机制,在四个粒度层级上联合分配稀疏性。
  • 在ImageNet上将DeiT-Base压缩至7.4G MACs,精度达80.88%。
  • 无需重要性评估或微调,可直接在常规算子上加速,实测吞吐提升超90%。

视觉Transformer计算开销大,严重限制其在资源受限硬件上的部署。现有结构化剪枝方法通常仅在单一粒度下操作,依赖多阶段流程、重要性排序、辅助求解器或后处理阈值,并需额外微调恢复精度。我们提出分层自动剪枝(HiAP),将ViT剪枝建模为单个预算感知的学习问题,一次性在四个粒度层级上联合分配稀疏性。HiAP在宏观层面(注意力头与前馈块)和微观层面(头内维度与前馈神经元)引入随机Gumbel-Sigmoid门控,与任务损失及解析的MAC成本项共同训练。预算系数引导网络达到目标计算量,门控在收敛时逐渐硬化为稀疏子网络。无需重要性启发式、排序指标、辅助求解器或二次微调。在ImageNet上,HiAP将DeiT-Base压缩至7.4G MACs,Top-1精度达80.88%;DeiT-Small压缩至3.1G MACs,精度79.33%,性能优于更复杂流水线且计算量相当。剪枝后的网络可原生通过标准算子加速,实测吞吐量减少超过90%的理论MAC开销。

原文摘要 · Abstract (English)

Vision Transformers require significant computational resources and memory bandwidth, severely limiting their deployment on resource-constraint hardware. Most structured pruning methods reduce theoretical cost effectively, yet they typically operate at a single structural granularity and depend on multi-stage pipelines with importance ranking, auxiliary solvers or post-hoc magnitude thresholding, followed by a separate fine-tuning phase to recover accuracy. We propose Hierarchical Auto-Pruning (HiAP), which casts ViT pruning as a single budget-aware learning problem and jointly allocates sparsity across four granularities in one end-to-end phase. HiAP introduces stochastic Gumbel-Sigmoid gates at macro level (attention heads and FFN blocks) and micro level (intra-head dimensions and FFN neurons), and trains them against the task loss together with an analytical MAC cost term. The budget coefficient steers the network to a target compute level while the gates gradually harden into a dense, smaller sub-network at convergence. It does not require importance heuristics, ranking metrics, auxiliary solvers or secondary fine-tuning. On ImageNet, HiAP compresses DeiT-Base to 7.4G MACs at 80.88% top-1 and DeiT-Small to 3.1G at 79.33%, competitive with substantially more complex pipelines at matched compute. The structurally pruned network can be accelerated natively on stock kernels, and more than 90% of the theoretical MAC reduction is realized as measured throughput on an A100.

视觉Transformer模型剪枝自动压缩高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。