提出混合精度量化方法,让低比特扩散模型性能大幅回升。
MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models
- 用峰度识别关键权重通道,动态分配不同精度位宽
- 在W2A4下FID降低58%,其他方法完全崩溃
- 适合部署在资源受限设备的低比特扩散模型
扩散模型在生成任务中广受关注,但计算开销大,难以在资源受限场景应用。量化通过降低参数位宽可显著节省存储与计算,但现有方法在极低比特(2-4比特)下仍导致性能严重下降,主要因激活值离散化加剧,候选值过少不利于关键通道量化,且离散特征阻碍扩散模型各时间步间的稳定学习。本文提出MPQ-DM,一种针对扩散模型的混合精度量化方法,包含两项关键技术:(1) 提出基于峰度的异常通道驱动混合量化(OMQ),量化异常显著通道并优化层内混合精度位宽分配以恢复精度;(2) 构建时间平滑关系蒸馏(TRD),将量化模型与全精度模型的离散与连续潜在表示映射至统一关系空间,减少跨时间步表征不一致。大量实验表明,相比最先进方法,MPQ-DM在极低比特下实现显著性能提升,在W2A4设置下FID降低58%,而其他方法均崩溃。
原文摘要 · Abstract (English)
Diffusion models have received wide attention in generation tasks. However, the expensive computation cost prevents the application of diffusion models in resource-constrained scenarios. Quantization emerges as a practical solution that significantly saves storage and computation by reducing the bit-width of parameters. However, the existing quantization methods for diffusion models still cause severe degradation in performance, especially under extremely low bit-widths (2-4 bit). The primary decrease in performance comes from the significant discretization of activation values at low bit quantization. Too few activation candidates are unfriendly for outlier significant weight channel quantization, and the discretized features prevent stable learning over different time steps of the diffusion model. This paper presents MPQ-DM, a Mixed-Precision Quantization method for Diffusion Models. The proposed MPQ-DM mainly relies on two techniques:(1) To mitigate the quantization error caused by outlier severe weight channels, we propose an Outlier-Driven Mixed Quantization (OMQ) technique that uses $Kurtosis$ to quantify outlier salient channels and apply optimized intra-layer mixed-precision bit-width allocation to recover accuracy performance within target efficiency.(2) To robustly learn representations crossing time steps, we construct a Time-Smoothed Relation Distillation (TRD) scheme between the quantized diffusion model and its full-precision counterpart, transferring discrete and continuous latent to a unified relation space to reduce the representation inconsistency. Comprehensive experiments demonstrate that MPQ-DM achieves significant accuracy gains under extremely low bit-widths compared with SOTA quantization methods. MPQ-DM achieves a 58\% FID decrease under W2A4 setting compared with baseline, while all other methods even collapse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。