用深度卷积替换部分注意力头，加速视觉大模型推理。
Accelerating Vision Foundation Models with Drop-in Depthwise Convolution
- 用深度卷积替代部分注意力头，实现即插即用加速。
- 图像分类与分割任务中推理速度提升17%-20%，性能损失极小。
- 适合资源受限设备部署，无需重新训练即可应用。
预训练视觉基础模型在少量微调下即可在多任务中表现优异，但其视觉变压器（ViT）骨干网络带来高推理开销，限制了在资源受限设备上的部署。本文通过挖掘部分注意力头的内在卷积特性，提出一种高效的深度卷积层，作为这些头的即插即用替代方案。同时，设计简单策略识别可替换头，并引入微调流程恢复下游任务性能。在图像分类与分割任务上，该方法实现17%-20%的推理加速，性能损失极小。通过详细推导、大量实验与效率基准验证了方法有效性，参考实现已公开。
原文摘要 · Abstract (English)
Pretrained vision foundation models deliver strong performance across tasks with limited fine-tuning. However, their Vision Transformer (ViT) backbones impose high inference costs, limiting deployment on resource-constrained devices. In this work, we accelerate large-scale pretrained ViTs while preserving their feature extraction capabilities by exploiting the intrinsic convolution-like behavior of some attention heads. Specifically, we introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for these heads. Additionally, we propose simple strategies to identify which heads can be replaced and introduce a fine-tuning procedure that recovers downstream task performance. Across both image classification and segmentation tasks, our method achieves 17-20\% percent inference speedup with minimal performance degradation. We validate the approach through detailed derivations, extensive experiments, and efficiency benchmarks. The reference implementation is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。