让ViT像可伸缩的弹簧一样灵活变小，适应不同设备算力。
Slicing Vision Transformer for Flexible Inference
- 通过激活多个子网络，实现单模型多尺寸推理。
- 仅一次训练即达独立训练效果，图像识别准确率提升1.6%。
- 适合移动端、边缘设备等资源受限场景使用。
视觉变压器（ViT）具有良好的可扩展性。本文旨在将大型ViT缩小以适应动态变化的资源约束环境。我们观察到较小的ViT本质上是较大ViT在不同宽度下的子网络。因此，提出一种名为Scala的通用框架，使单个网络能够表示多个不同尺寸的ViT，并支持灵活推理，契合ViT按宽度调整的内在设计。具体而言，Scala在训练中激活多个子网络，引入隔离激活机制以解耦最小子网络与其他子网络的关系，并采用尺度协调机制确保每个子网络获得简化、稳定且准确的学习目标。在多种任务上的全面实验证明，仅需一次训练，Scala即可学习可剪枝的表示，无需修改原始ViT结构，性能媲美独立训练。相比现有方法，Scala在ImageNet-1K上平均提升1.6%，且参数更少。
原文摘要 · Abstract (English)
Vision Transformers (ViT) is known for its scalability. In this work, we target to scale down a ViT to fit in an environment with dynamic-changing resource constraints. We observe that smaller ViTs are intrinsically the sub-networks of a larger ViT with different widths. Thus, we propose a general framework, named Scala, to enable a single network to represent multiple smaller ViTs with flexible inference capability, which aligns with the inherent design of ViT to vary from widths. Concretely, Scala activates several subnets during training, introduces Isolated Activation to disentangle the smallest sub-network from other subnets, and leverages Scale Coordination to ensure each sub-network receives simplified, steady, and accurate learning objectives. Comprehensive empirical validations on different tasks demonstrate that with only one-shot training, Scala learns slimmable representation without modifying the original ViT structure and matches the performance of Separate Training. Compared with the prior art, Scala achieves an average improvement of 1.6% on ImageNet-1K with fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。