通过增加全局令牌宽度，让视觉变压器更快更准。
Thicker and Quicker: A Jumbo Token for Fast Plain Vision Transformers
- 引入宽大全局令牌，保持纯视觉变压器结构。
- 在ImageNet-1K上速度提升0.1%-13%，精度不降。
- 兼容预训练与动态裁剪，适合实用部署场景。
视觉变压器（ViTs）通用且准确，但速度慢，在效率关键场景下不够实用。现有加速方法多采用非ViT混合架构，牺牲通用性，或压缩令牌，影响精度。许多非ViT架构虽快且准，但无法像ViT一样处理任意输入形状、支持最先进自监督预训练、通过丢弃令牌降低计算量等。本文通过缩小局部补丁令牌宽度，同时增加一个全新的宽大全局令牌（Jumbo token），使模型变快。该全局令牌由更宽的前馈网络（FFN）处理，提升模型容量，而其仅处理单个令牌，参数跨层共享，兼顾速度与内存效率。关键在于，该令牌仅使用注意力机制，非分层设计，保持与纯ViT一致的简单性、可扩展性与灵活性。实验表明，从纳米到大型规模，Jumbo在保持ImageNet-1K吞吐率不变的前提下，优于基础ViT（含Registers）；在ADE20K分割任务中提升1.9%-3.1%；在MAE预训练线性探测中提升4.9%；在ImageNet-C测试时适应任务中提升5.2%；在时间序列建模中表现优异。其速度-精度权衡优于专用高效非ViT模型，同时保持对传统ViT方法的兼容性，利于实际应用。代码与权重已公开：https://github.com/antofuller/jumbo
原文摘要 · Abstract (English)
ViTs are general and accurate, and address many tasks, but ViTs are slow, and are not always practical when efficiency is key. Existing methods for faster ViTs design hybrid non-ViT architectures, losing generality, or shrink their tokens, sacrificing accuracy. Many non-ViT architectures are both fast and accurate. Yet, without significant modifications, they cannot do what ViTs can: process other input shapes, pre-train by SOTA self-supervised learning, reduce computation by dropping tokens, and more. We make ViTs faster by reducing patch token width while increasing global token width by adding a new Jumbo token. Our wider Jumbo token is processed by its own wider FFN to increase model capacity. Yet our Jumbo FFN is efficient: it processes a single token, for speed, and its parameters are shared across all layers, for memory. Crucially, our Jumbo is attention-only and non-hierarchical, like a plain ViT, so it is simple, scalable, flexible, and compatible with ViT methods new and old. Jumbo improves over ViT baselines with Registers from Nano to Large scales while maintaining speed/throughput on ImageNet-1K (0.1-13%). Jumbo also improves segmentation (1.9-3.1% on ADE20K), MAE pre-training (4.9% linear probing on ImageNet-1K), test-time adaptation (5.2% on ImageNet-C), and time series modeling. Our Jumbo models even achieve better speed-accuracy trade-offs than specialized non-ViT compute-efficient models, while maintaining plain-ViT compatibility for practicality. Code and weights are available: https://github.com/antofuller/jumbo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。