轻量级混合视觉模型实现高效多任务适配，适合移动端部署。
PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context
- 通过参数高效适配技术实现共享主干网络的多任务学习。
- 在少于100M参数下性能超越传统ViT适配方法。
- 结合剪枝技术，兼顾精度与存储效率，适合移动设备。
随着变换器在自然语言处理中的成功，其在计算机视觉领域也逐渐普及。尽管变换器表现优异且具备良好的多任务能力，但高计算需求限制了其在资源受限场景的应用。因此，许多低资源应用仍依赖卷积或混合模型，在100M参数以下取得最佳效果。与此同时，能够通过共享变换器主干实现多下游任务的高效适配技术尚未被应用于混合变换器。本文提出PETAH：面向混合变换器的参数高效任务适配方法，并结合剪枝技术，构建高性能且存储友好的多任务模型。在图像分类及其他视觉任务上的广泛评估表明，所提PETAH适配的混合模型在参数更少、移动端更高效的同时，优于现有ViT任务适配方法。
原文摘要 · Abstract (English)
Following their success in natural language processing (NLP), there has been a shift towards transformer models in computer vision. While transformers perform well and offer promising multi-tasking performance, due to their high compute requirements, many resource-constrained applications still rely on convolutional or hybrid models that combine the benefits of convolution and attention layers and achieve the best results in the sub 100M parameter range. Simultaneously, task adaptation techniques that allow for the use of one shared transformer backbone for multiple downstream tasks, resulting in great storage savings at negligible cost in performance, have not yet been adopted for hybrid transformers. In this work, we investigate how to achieve the best task-adaptation performance and introduce PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers. We further combine PETAH adaptation with pruning to achieve highly performant and storage friendly models for multi-tasking. In our extensive evaluation on classification and other vision tasks, we demonstrate that our PETAH-adapted hybrid models outperform established task-adaptation techniques for ViTs while requiring fewer parameters and being more efficient on mobile hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。