arXiv:2501.07161cs.AI2025-01中稿 · Future Generation …被引 7

QuantuneV2通过编译器级量化提升嵌入式AI模型性能,兼顾精度与速度。

QuantuneV2: Compiler-Based Local Metric-Driven Mixed Precision Quantization for Practical Embedded AI Applications

  • 基于局部度量的混合精度量化,仅需两次推理完成优化。
  • 相比现有方法,精度提升10.28%,推理速度加快12.52%。
  • 适合资源受限的嵌入式部署场景,计算开销低。

混合精度量化方法旨在减少模型大小的同时最小化精度损失。然而,现有研究通常需要重新训练,并未考虑编译过程中的计算开销和中间表示(IR),限制了其在编译器层面的应用。该计算开销指推理过程中频繁进行量化与反量化操作所导致的运行时延迟,尤其在逐算子级别执行时影响显著。为此,我们提出QuantuneV2,一种面向实际嵌入式AI应用的编译器级混合精度量化方法。QuantuneV2仅需两次推理(一次量化前,一次量化后),计算复杂度为O(n),随参数量线性增长。通过引入权重、激活值、信噪比(SQNR)与均方误差(MSE)等局部度量,提升了敏感性分析的稳定性。同时,通过选择最优中间表示并使用算子融合,有效降低了计算开销。实验结果表明,在五个模型(ResNet18v1、ResNet50v1、SqueezeNetv1、VGGNet、MobileNetv2)上,QuantuneV2相较现有方法最高实现10.28%的精度提升和12.52%的加速效果。这表明QuantuneV2在保持高计算效率的同时显著提升模型表现,适用于嵌入式AI环境部署。

原文摘要 · Abstract (English)

Mixed-precision quantization methods have been proposed to reduce model size while minimizing accuracy degradation. However, existing studies require retraining and do not consider the computational overhead and intermediate representations (IR) generated during the compilation process, limiting their application at the compiler level. This computational overhead refers to the runtime latency caused by frequent quantization and dequantization operations during inference. Performing these operations at the individual operator level causes significant runtime delays. To address these issues, we propose QuantuneV2, a compiler-based mixed-precision quantization method designed for practical embedded AI applications. QuantuneV2 performs inference only twice, once before quantization and once after quantization, and operates with a computational complexity of O(n) that increases linearly with the number of model parameters. We also made the sensitivity analysis more stable by using local metrics like weights, activation values, the Signal to Quantization Noise Ratio, and the Mean Squared Error. We also cut down on computational overhead by choosing the best IR and using operator fusion. Experimental results show that QuantuneV2 achieved up to a 10.28 percent improvement in accuracy and a 12.52 percent increase in speed compared to existing methods across five models: ResNet18v1, ResNet50v1, SqueezeNetv1, VGGNet, and MobileNetv2. This demonstrates that QuantuneV2 enhances model performance while maintaining computational efficiency, making it suitable for deployment in embedded AI environments.

量化嵌入式AI编译器优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。