用任务相关性自动选关键图像令牌，提速1.5到4倍且性能损失极小。
Token Cropr: Faster ViTs for Quite a Few Tasks
- 训练时加辅助头，端到端学习哪些图像令牌重要。
- 推理时移除辅助头，速度接近随机剪枝，但性能更好。
- 在分类、分割、检测等任务上均有效，适合移动端部署。
视觉变换器（ViTs）在资源受限场景下的应用需要提升推理吞吐量。现有令牌剪枝与合并方法虽能降低令牌数量，但难以兼顾速度、性能与多任务通用性。本文提出一种新剪枝方法：通过辅助预测头在训练中学习基于任务相关性的令牌选择，训练完成后移除辅助头，推理速度接近随机剪枝，却保持更高性能。我们在图像分类、语义分割、目标检测和实例分割任务上评估该方法，在多个数据集上实现1.5至4倍的加速，性能下降极小。以ADE20k语义分割为例，相比无剪枝基线获得2倍提速，5次种子测试下中位mIoU仅下降0.1。
原文摘要 · Abstract (English)
The adoption of Vision Transformers (ViTs) in resource-constrained applications necessitates improvements in inference throughput. To this end several token pruning and merging approaches have been proposed that improve efficiency by successively reducing the number of tokens. However, it remains an open problem to design a token reduction method that is fast, maintains high performance, and is applicable to various vision tasks. In this work, we present a token pruner that uses auxiliary prediction heads that learn to select tokens end-to-end based on task relevance. These auxiliary heads can be removed after training, leading to throughput close to that of a random pruner. We evaluate our method on image classification, semantic segmentation, object detection, and instance segmentation, and show speedups of 1.5 to 4x with small drops in performance. As a best case, on the ADE20k semantic segmentation benchmark, we observe a 2x speedup relative to the no-pruning baseline, with a negligible performance penalty of 0.1 median mIoU across 5 seeds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。