针对视觉自回归模型的高精度量化难题，提出软硬件协同优化方案，显著提升边缘设备部署性能。
FPQVAR: Floating Point Quantization for Visual Autoregressive Model with FPGA Hardware Co-design
- 采用双格式量化与分组哈达玛变换应对激活值分布不均和通道异常波动
- 4比特量化下FID降至3.58，IS提升至241.5，逼近全精度模型表现
- 基于FPGA的加速器实现1.1图像/秒吞吐，能效比超整数加速器3.6倍
视觉自回归（VAR）建模通过从粗到细的多尺度预测，实现了图像生成范式革新，相比扩散模型具备更高画质和更快推理速度。但其庞大参数量与计算开销限制了在边缘设备的部署。为此，本文提出面向VAR的高效后训练浮点量化框架FPQVAR，实现算法与硬件协同设计。算法层面，针对激活值分布极不均衡问题，提出双格式量化；针对时变异常通道，提出分组哈达玛变换与可学习变换机制。硬件层面，首次在FPGA上设计低比特浮点量化器与查找表乘法器，并构建支持低比特浮点计算的两级流水线加速器。实验表明，在4比特量化下，FID从10.83降至3.58，Inception Score从175.9提升至241.5；6比特量化性能已达FP16模型水平。基于AMD-Xilinx VCK190 FPGA的加速器达到1.1图像/秒吞吐，是整数基加速器的3.1倍，能效分别提升3.6倍（相较整数基）和2.8倍（相较GPU）。
原文摘要 · Abstract (English)
Visual autoregressive (VAR) modeling has marked a paradigm shift in image generation from next-token prediction to next-scale prediction. VAR predicts a set of tokens at each step from coarse to fine scale, leading to better image quality and faster inference speed compared to existing diffusion models. However, the large parameter size and computation cost hinder its deployment on edge devices. To reduce the memory and computation cost, we propose FPQVAR, an efficient post-training floating-point (FP) quantization framework for VAR featuring algorithm and hardware co-design. At the algorithm level, we first identify the challenges of quantizing VAR. To address them, we propose Dual Format Quantization for the highly imbalanced input activation. We further propose Group-wise Hadamard Transformation and GHT-Aware Learnable Transformation to address the time-varying outlier channels. At the hardware level, we design the first low-bit FP quantizer and multiplier with lookup tables on FPGA and propose the first FPGA-based VAR accelerator featuring low-bit FP computation and an elaborate two-level pipeline. Extensive experiments show that compared to the state-of-the-art quantization method, our proposed FPQVAR significantly improves Fréchet Inception Distance (FID) from 10.83 to 3.58, Inception Score (IS) from 175.9 to 241.5 under 4-bit quantization. FPQVAR also significantly improves the performance of 6-bit quantized VAR, bringing it on par with the FP16 model. Our accelerator on AMD-Xilinx VCK190 FPGA achieves a throughput of 1.1 image/s, which is 3.1x higher than the integer-based accelerator. It also demonstrates 3.6x and 2.8x higher energy efficiency compared to the integer-based accelerator and GPU baseline, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。