为扩散模型设计混合精度量化，加速采样并提升图像质量。
MPQ-Diff: Mixed Precision Quantization for Diffusion Models
- 按层和采样步数动态分配权重与激活的位宽
- 在LSUN和ImageNet上FID分别降至15.39和14.93
- 用网络正交性指标衡量层重要性，降低调优开销
扩散模型通过随机去噪过程生成高质量图像，但采样时间较长。对训练好的扩散模型进行固定位宽后量化（如权重4比特、激活8比特）可有效加速采样且保持图像质量。受各层跨层依赖性随层和采样步数变化的启发，本文提出混合精度量化方案MPQ-Diff，为不同层的权重与激活分配不同位宽。我们以网络正交性度量作为代理指标，评估每层在各采样步中的相对重要性，并采用均匀采样策略避免全时间步正交性估计带来的过高开销。在LSUN和ImageNet数据集上的实验表明，相比固定精度量化，FID得分分别从65.73降至15.39，从52.66降至14.93，显著提升生成质量。
原文摘要 · Abstract (English)
Diffusion models (DMs) generate remarkable high quality images via the stochastic denoising process, which unfortunately incurs high sampling time. Post-quantizing the trained diffusion models in fixed bit-widths, e.g., 4 bits on weights and 8 bits on activation, is shown effective in accelerating sampling time while maintaining the image quality. Motivated by the observation that the cross-layer dependency of DMs vary across layers and sampling steps, we propose a mixed precision quantization scheme, MPQ-Diff, which allocates different bit-width to the weights and activation of the layers. We advocate to use the cross-layer correlation of a given layer, termed network orthogonality metric, as a proxy to measure the relative importance of a layer per sampling step. We further adopt a uniform sampling scheme to avoid the excessive profiling overhead of estimating orthogonality across all time steps. We evaluate the proposed mixed-precision on LSUN and ImageNet, showing a significant improvement in FID from 65.73 to 15.39, and 52.66 to 14.93, compared to their fixed precision quantization, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。