通过分层分组量化与SVD引导的混合精度，实现低比特推理的高效加速。
SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precision
- 将敏感组件分离至高精度低秩路径，其余部分用低比特分组量化处理。
- 在ViT-Base和Llama2-7B上分别达到12.7TOPS/W和13.4TOPS/W能效峰值。
- 适合部署在资源受限设备上的Transformer低比特推理应用。
低比特量化可有效降低Transformer推理的计算与内存开销，但激进的位宽压缩因激活值异常值导致准确率下降。现有方法如异常值处理和分组量化虽保持高精度，但能耗较高。为此，我们提出SeVeDo，一种基于SVD的异构加速器，将敏感组件结构化分离至高精度低秩路径，其余计算在低比特残差数据路径中完成，结合分组量化。进一步采用分层分组量化（HGQ），融合粗粒度浮点缩放与细粒度移位，有效降低反量化成本。同时，SVD引导的混合精度（SVD-MP）通过低秩分解静态分配更高位宽给敏感组件，最小化浮点运算开销。实验表明，SeVeDo峰值能效达13.8TOPS/W，优于传统设计，在ViT-Base和Llama2-7B上分别实现12.7TOPS/W和13.4TOPS/W。
原文摘要 · Abstract (English)
Low-bit quantization is a promising technique for efficient transformer inference by reducing computational and memory overhead. However, aggressive bitwidth reduction remains challenging due to activation outliers, leading to accuracy degradation. Existing methods, such as outlier-handling and group quantization, achieve high accuracy but incur substantial energy consumption. To address this, we propose SeVeDo, an energy-efficient SVD-based heterogeneous accelerator that structurally separates outlier-sensitive components into a high-precision low-rank path, while the remaining computations are executed in a low-bit residual datapath with group quantization. To further enhance efficiency, Hierarchical Group Quantization (HGQ) combines coarse-grained floating-point scaling with fine-grained shifting, effectively reducing dequantization cost. Also, SVD-guided mixed precision (SVD-MP) statically allocates higher bitwidths to precision-sensitive components identified through low-rank decomposition, thereby minimizing floating-point operation cost. Experimental results show that SeVeDo achieves a peak energy efficiency of 13.8TOPS/W, surpassing conventional designs, with 12.7TOPS/W on ViT-Base and 13.4TOPS/W on Llama2-7B benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。