用小模型指导大模型丢弃无用图像片段，大幅降低计算量。
TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
- 用轻量级模型评估图像块重要性，决定哪些可丢弃。
- 在不降精度前提下，最高节省80%计算量（FLOPs）。
- 无需修改结构，适配多种ViT模型，部署简单。
视觉变压器（ViTs）在图像分类中表现强劲，但处理所有图像块带来高昂计算成本。为在不牺牲准确率的前提下降低大型ViT的推理开销，我们提出TinyDrop——一种无需训练、由轻量视觉模型引导的令牌丢弃框架。该框架在推理时通过小型引导模型评估各图像块的重要性，对低重要性块进行选择性丢弃，从而减少注意力计算量。该方法即插即用，无需架构改动，兼容多种ViT结构。在标准图像分类基准上的实验表明，该框架可使ViT的计算量（FLOPs）最多降低80%，同时保持极小精度损失，展现出良好的泛化能力与实用价值。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) achieve strong performance in image classification but incur high computational costs from processing all image tokens. To reduce inference costs in large ViTs without compromising accuracy, we propose TinyDrop, a training-free token dropping framework guided by a lightweight vision model. The guidance model estimates the importance of tokens while performing inference, thereby selectively discarding low-importance tokens if large vit models need to perform attention calculations. The framework operates plug-and-play, requires no architectural modifications, and is compatible with diverse ViT architectures. Evaluations on standard image classification benchmarks demonstrate that our framework reduces FLOPs by up to 80% for ViTs with minimal accuracy degradation, highlighting its generalization capability and practical utility for efficient ViT-based classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。