FlexViT用可重构FPGA加速边缘视觉Transformer，支持混合层型高效推理。
FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers
- 统一张量计算引擎，动态切换数据流模式适应不同层结构。
- 单次遍历完成累加，减少内存带宽需求，实测加速比达1.40倍。
- 适合资源受限的边缘设备部署混合架构的视觉Transformer模型。
在边缘平台部署视觉变压器（ViT）模型仍具挑战性，因其高计算需求及现代混合型ViT模型中全连接与卷积层并存带来的架构异构性，导致张量形状变化大，需灵活高效的FPGA加速。本文提出FlexViT，一种面向资源受限边缘设备的可重构FPGA加速器，用于高效执行ViT推理。基于SECDA-TFLite框架，采用软硬件协同设计，将全连接与卷积层均映射至统一的高吞吐量INT8 GEMM引擎，通过运行时im2col转换实现。为支持多样化的层配置，提出双模式数据流，在运行时重构计算阵列以动态切换输入与权重重用。进一步引入深度优先分块策略，单次遍历完成累加，消除片外部分和传输，降低内存带宽需求。我们在PYNQ-Z2 FPGA上实现FlexViT，评估了代表性的一组ViT模型。结果表明，加速器执行层最高获得2.74倍加速，端到端相比纯CPU执行最高提升1.40倍。代码已开源：https://github.com/gicLAB/FlexViT
原文摘要 · Abstract (English)
Deploying Vision Transformer (ViT) models on edge platforms remains challenging due to their high computational demands and the architectural heterogeneity of modern hybrid ViT models, which incorporate both fully connected and convolutional layers. This heterogeneity leads to significant variation in tensor shapes, requiring flexible and efficient FPGA-based acceleration. In this paper, we present FlexViT, a reconfigurable FPGA accelerator for efficient ViT inference on resource-constrained edge devices. Built on the SECDA-TFLite framework, FlexViT employs a hardware-software co-design approach that maps both fully connected and convolutional layers onto a unified high-throughput INT8 GEMM engine using a runtime im2col transformation. To efficiently support diverse layer configurations, we propose a dual-mode dataflow that dynamically switches between input and weight reuse by reconfiguring the compute array at runtime. We further introduce a depth-first tiling strategy that completes accumulation in a single pass, eliminating off-chip partial-sum transfers and reducing memory bandwidth requirements. We implement FlexViT on a PYNQ-Z2 FPGA and evaluate it across a representative set of ViT models. FlexViT achieves up to 2.74x speedup on accelerator-executed layers, translating into up to 1.40x end-to-end speedup compared to CPU-only execution. The code is available at: https://github.com/gicLAB/FlexViT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。