首个可同时动态调整深度、宽度和注意力数量的Transformer模型。
AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens
- 统一支持深度、宽度、注意力数三轴自适应，单模型适配多场景。
- 图像分类准确率85.4%，吞吐量比FlexiViT-L高36%。
- 适合边缘设备部署，可在降低24%-33%计算量时保持精度稳定。
现代Transformer在各类任务中表现卓越，但推理时计算分配僵化。实际部署常需适配不同硬件与延迟要求，而现有动态计算方法多仅关注单一维度（如减少注意力数）。本文提出AdaPerceiver，首个在单一模型中实现深度、宽度与注意力数三轴统一自适应的Transformer架构。通过高效联合训练策略，确保各配置下性能稳定。在图像分类、语义分割与深度估计任务上评估：图像分类中，其准确率达85.4%，吞吐量比FlexiViT-L高36%；在密集预测任务中，达到ViT-H/14精度，但编码器浮点运算量减少约26倍；结合策略后，可使图像分类任务减少24%-33%浮点运算量，同时保持ImageNet1K精度±0.1个百分点。
原文摘要 · Abstract (English)
Modern transformer architectures achieve remarkable performance across tasks and domains but remain rigid in how they allocate computation at inference time. Real-world deployment often requires models to adapt to diverse hardware and latency constraints, yet most approaches to dynamic computation focus on a single axis -- such as reducing the number of tokens. We present a novel capability: AdaPerceiver, the first transformer architecture with unified adaptivity across depth, width, and tokens within a single model. We propose an architecture that supports adaptivity along these axes. We couple this with an efficient joint training regime that ensures the model maintains performance across its various configurations. We evaluate AdaPerceiver on image classification, semantic segmentation, and depth estimation tasks. On image classification, AdaPerceiver expands the accuracy-throughput Pareto front. It achieves 85.4% accuracy while yielding 36% higher throughput than FlexiViT-L. On dense prediction, AdaPerceiver matches ViT-H/14 while having $\sim$26x fewer encoder FLOPs (floating-point operations) on semantic segmentation and depth estimation. Finally, we show how AdaPerceiver equipped with a policy can maintain ImageNet1K accuracy ($\pm0.1$ percentage points) while reducing FLOPs by $24-33$%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。