arXiv:2602.06300cs.CVcs.AI2026-02

将ViT转换为可适配BPU的卷积结构,实现3.8倍加速且精度损失极小。

Accelerating Vision Transformers on Brain Processing Unit

  • 用卷积操作重构ViT线性层与归一化,适配BPU硬件
  • 量化DeiT-Base在ImageNet上达80.4%准确率,速度提升3.8倍
  • 无需重训练即可部署,适合边缘设备上的ViT加速

随着深度学习发展,专用神经处理硬件如脑处理器(Brain Processing Units, BPUs)应运而生,其针对卷积操作优化了INT8计算能力。与此同时,视觉变换器(Vision Transformer, ViT)模型如数据高效图像变换器(DeiT)展现出优异性能,在计算机视觉中作用日益重要。然而,由于CNN优化硬件与ViT计算特性存在架构不匹配——即变换器中的线性层处理三维数据,而BPU加速设计针对四维卷积操作——导致在部署ViT时难以利用BPU优势。为此,我们提出一种新方法:通过将线性层和层归一化操作替换为精心设计的卷积算子,重构ViT结构。该方法使DeiT能充分使用BPU加速能力,同时保留原始权重参数,无需重新训练或微调。据我们所知,这是首个成功实现完整BPU加速的ViT部署。在ImageNet分类数据集上,量化后的DeiT-Base模型达到80.4%准确率(原为81.8%),推理速度提升最高达3.8倍。在花卉分类数据集上,微调后的DeiT模型仅损失0.5%准确率,进一步验证方法有效性。

原文摘要 · Abstract (English)

With the advancement of deep learning technologies, specialized neural processing hardware such as Brain Processing Units (BPUs) have emerged as dedicated platforms for CNN acceleration, offering optimized INT8 computation capabilities for convolutional operations. Meanwhile, Vision Transformer (ViT) models, such as the Data-efficient Image Transformer (DeiT), have demonstrated superior performance and play increasingly crucial roles in computer vision tasks. However, due to the architectural mismatch between CNN-optimized hardware and Vision Transformer computation characteristics--namely, that linear layers in Transformers operate on three-dimensional data while BPU acceleration is designed for four-dimensional convolution operations-it is difficult or even impossible to leverage BPU's advantages when deploying Vision Transformers. To address this challenge, we propose a novel approach that restructures the Vision Transformer by replacing linear layers and layer normalization operations with carefully designed convolutional operators. This enables DeiT to fully utilize the acceleration capabilities of BPUs, while allowing the original weight parameters to be inherited by the restructured models without retraining or fine-tuning. To the best of our knowledge, this is the first successful deployment of Vision Transformers that fully leverages BPU classification datasets demonstrate the effectiveness of our approach. Specifically, the quantized DeiT-Base model achieves 80.4% accuracy on ImageNet, compared to the original 81.8%, while obtaining up to a 3.8* inference speedup. Our finetuned DeiT model on the flower classification dataset also achieves excellent performance, with only a 0.5% accuracy drop for the DeiT-Base model, further demonstrating the effectiveness of our method.

ViT加速BPU模型转换边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。