针对边缘设备优化视觉马尔可夫模型推理,实现4.96倍加速和59.8倍能效提升。
ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGA

- 提出硬件感知量化方法,结合动态激活与通道平滑,解决低比特下权重分布失真问题。
- 在FPGA上实现4比特分块加性幂次权重量化,支持运行时配置以适配不同模型规模。
- 设计流水线化状态空间引擎与查表单元,匹配边缘设备数据流需求,适合资源受限场景部署。
视觉马尔可夫(ViM)模型通过状态空间模型(SSMs)的线性复杂度,在效率上优于Transformer,但其在FPGA上的高效部署仍具挑战。线性层对动态激活异常值敏感,导致静态量化失效;而均匀量化在低比特下难以捕捉权重分布。此外,尽管关联扫描可加速GPU上的SSM计算,但其内存访问模式与FPGA所需的流式数据流不匹配。为此,本文提出ViM-Q,一种面向边缘端的可扩展算法-硬件协同设计。引入硬件感知量化方案,结合动态逐令牌激活量化与逐通道平滑,缓解异常值影响;同时采用自定义的4比特每块加性幂次(APoT)权重量化。模型部署于可运行时参数化的FPGA加速器,包含基于查找表(LUT)单元的线性引擎(以移位加替代乘法),以及细粒度流水线化的SSM引擎,可并行状态维度且保持序列递归特性。关键优势在于硬件支持运行时配置,适应ViM系列中多样维度与输入分辨率。在AMD ZCU102 FPGA上,对于低批次的ViM-tiny推理,相较量化版NVIDIA RTX 3090 GPU基线,平均实现4.96倍加速与59.8倍能效提升。该协同设计为在资源受限边缘设备上部署ViM模型提供了可行路径。
原文摘要 · Abstract (English)
Vision Mamba (ViM) models offer a compelling efficiency advantage over Transformers by leveraging the linear complexity of State Space Models (SSMs), yet efficiently deploying them on FPGAs remains challenging. Linear layers struggle with dynamic activation outliers that render static quantization ineffective, while uniform quantization fails to capture the weight distribution at low bit-widths. Furthermore, while associative scan accelerates SSMs on GPUs, its memory access patterns are misaligned with the streaming dataflow required by FPGAs. To address these challenges, we present ViM-Q, a scalable algorithm-hardware co-design for end-to-end ViM inference on the edge. We introduce a hardware-aware quantization scheme combining dynamic per-token activation quantization and per-channel smoothing to mitigate outliers, alongside a custom 4-bit per-block Additive Power-of-Two (APoT) weight quantization. The models are deployed on a runtime-parameterizable FPGA accelerator featuring a linear engine employing a Lookup-Table (LUT) unit to replace multiplications with shift-add operations, and a fine-grained pipelined SSM engine that parallelizes the state dimension while preserving sequential recurrence. Crucially, the hardware supports runtime configuration, adapting to diverse dimensions and input resolutions across the ViM family. Implemented on an AMD ZCU102 FPGA, ViM-Q achieves an average 4.96x speedup and 59.8x energy efficiency gain over a quantized NVIDIA RTX 3090 GPU baseline for low-batch inference on ViM-tiny. This co-design shows a viable path for deploying ViM models on resource-constrained edge devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。