根据图像内容自适应调整分块大小，提升ViT模型速度与效率
Accelerating Vision Transformers with Adaptive Patch Sizes
- 同一图像中使用多种分块尺寸，复杂区域用小块，平滑区域用大块
- 在ViT-L上提速40%，ViT-H上提速50%，训练和推理时间最多减少30%
- 可直接用于已微调的ViT模型，1个周期内即可快速收敛
视觉变换器（ViTs）对输入图像采用统一尺寸的分块，导致高分辨率图像输入序列过长。我们提出自适应分块变换器（APT），在同一图像中使用多种不同大小的分块。APT通过在较均匀区域分配较大分块、在较复杂区域使用较小分块，显著减少输入标记总数。APT在ViT推理和训练中实现显著加速，在ViT-L上提升吞吐量40%，在ViT-H上提升50%，同时保持下游任务性能。该方法可应用于已微调的ViT模型，仅需1个训练周期即可收敛。在高分辨率密集视觉任务中，显著降低训练与推理时间，视觉问答、目标检测和语义分割任务中最快提速30%。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) partition input images into uniformly sized patches regardless of their content, resulting in long input sequence lengths for high-resolution images. We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image. APT reduces the total number of input tokens by allocating larger patch sizes in more homogeneous areas and smaller patches in more complex ones. APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40% on ViT-L and 50% on ViT-H while maintaining downstream performance, and can be applied to a previously fine-tuned ViT, converging in as little as 1 epoch. It also significantly reduces training and inference time without loss of performance in high-resolution dense visual tasks, achieving up to 30\% faster training and inference in visual QA, object detection, and semantic segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。