arXiv:2409.13999cs.CV2024-09被引 1

让简单图像提前退出,大幅降低视觉Transformer推理成本。

Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer

  • 在ViT中设置多个输出层,简单样本早退出,节省计算
  • 相比主流方法,准确率更高且推理速度提升明显
  • 适合需要高效部署的视觉模型微调场景

参数高效迁移学习(PETL)在将大规模预训练的视觉Transformer(ViT)适配到下游任务方面展现出巨大潜力。现有方法主要聚焦于减少可学习参数数量,虽节省存储,但对简单样本仍分配过多计算资源,导致推理效率低下。为此,本文提出一种推理高效的微调方法——多出口微调(MET)。MET在预训练的ViT主干中集成多个出口,由于ViT的预测由线性分类器完成,每个出口配备一个线性预测头。推理时,简单样本在早期出口即可退出,仅困难样本流向最后一层,从而节省计算开销。MET包含出口专用适配器(E-adapters)和图正则化。E-adapters用于提取不同出口所需的表征,所有适配器共享相同的降维与升维矩阵以保证参数效率。考虑到线性分类器性能受样本间关系影响,采用图正则化优化早期出口输入表征。大量实验验证了MET的有效性,结果表明其在准确率与推理效率上均显著优于当前最优方法。

原文摘要 · Abstract (English)

Parameter-efficient transfer learning (PETL) has shown great potential in adapting a vision transformer (ViT) pre-trained on large-scale datasets to various downstream tasks. Existing studies primarily focus on minimizing the number of learnable parameters. Although these methods are storage-efficient, they allocate excessive computational resources to easy samples, leading to inefficient inference. To address this issue, we introduce an inference-efficient tuning method termed multiple-exit tuning (MET). MET integrates multiple exits into the pre-trained ViT backbone. Since the predictions in ViT are made by a linear classifier, each exit is equipped with a linear prediction head. In inference stage, easy samples will exit at early exits and only hard enough samples will flow to the last exit, thus saving the computational cost for easy samples. MET consists of exit-specific adapters (E-adapters) and graph regularization. E-adapters are designed to extract suitable representations for different exits. To ensure parameter efficiency, all E-adapters share the same down-projection and up-projection matrices. As the performances of linear classifiers are influenced by the relationship among samples, we employ graph regularization to improve the representations fed into the classifiers at early exits. Finally, we conduct extensive experiments to verify the performance of MET. Experimental results show that MET has an obvious advantage over the state-of-the-art methods in terms of both accuracy and inference efficiency.

视觉Transformer推理效率参数高效

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。