统一协调三种方法,让视觉Transformer更智能地省计算。
Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers

- 按合并-判断-裁剪顺序协同处理令牌,避免中间表示不稳定。
- 在ImageNet-1k上降低48%推理能耗,校准误差减少4倍。
- 无需微调即可适配不同数据集,支持运行时调节性能与速度平衡。
视觉变压器(Vision Transformers)虽在图像分类上表现优异,但对所有图像区域施加近似相同的计算量,即使部分区域冗余或无信息。现有自适应推理方法通过选择性压缩令牌或提前终止推理来降低成本,但组合这些机制常导致中间表示不稳定和准确率下降。我们提出Fusion,一种统一的自适应推理框架,通过简单的分阶段设计协调令牌合并、早期退出和令牌裁剪:先合并令牌,再评估置信度,最后仅对继续推理的样本进行裁剪。该顺序使三者协作而非竞争。Fusion还包含轻量级路由模块,可按输入动态调整压缩强度,并支持推理时无须重训练即可调节精度-延迟权衡。在DeiT-S与ImageNet-1k上,Fusion在相近计算预算下达到或超越当前最优自适应ViT方法,校准误差降低最多4倍,推理能耗减少48%。在ImageNet-100、CIFAR-100和ImageNette上使用多个ViT主干网络的实验表明,该方法具有稳定迁移能力,无需针对特定数据集调参。
原文摘要 · Abstract (English)
Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many regions are redundant or uninformative. Recent adaptive inference methods reduce this cost by selectively compressing tokens or terminating inference early, but combining these mechanisms often causes unstable intermediate representations and accuracy degradation. We introduce Fusion, a unified adaptive inference framework that coordinates token merging, early exiting, and token pruning through a simple staged design: tokens are merged first, confidence is evaluated next, and pruning is applied only to samples that continue inference. This ordering allows the three mechanisms to operate cooperatively rather than competitively. Fusion further includes lightweight routing modules that adapt compression strength to each input and support inference-time adjustment of the accuracy--latency trade-off without retraining. On ImageNet-1k with DeiT-S, Fusion matches or surpasses state-of-the-art adaptive ViT methods at comparable compute budgets while reducing calibration error by up to $4\times$ and inference energy by $48\%$. Experiments across ImageNet-100, CIFAR-100, and ImageNette with multiple ViT backbones demonstrate consistent transferability without dataset-specific tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。