针对扩散模型量化中时间步梯度不均的问题,提出加权校准新方法。
Gradient-Aligned Calibration for Post-Training Quantization of Diffusion Models
- 按时间步差异分配校准样本权重,使梯度对齐
- 在CIFAR-10等数据集上实现更优图像生成质量
- 适合需要高效部署扩散模型的工程师参考
扩散模型通过逐步估计从噪声高斯分布到真实图像的平滑转换,在图像生成中表现出色。然而,其实际部署受限于推理速度慢、内存占用高及噪声估计计算量大。后训练量化(PTQ)成为加速采样和降低内存开销的有前景方案。现有扩散模型的PTQ方法通常对各时间步的校准样本采用统一权重,但不同时间步的数据对扩散过程贡献不同,且激活分布与梯度随时间步变化,统一量化方式效果不佳。本文提出一种新方法,通过学习为校准样本分配最优权重,使量化模型在各时间步的梯度方向一致,从而提升量化性能。在CIFAR-10、LSUN-Bedrooms和ImageNet上的大量实验表明,该方法优于其他扩散模型的PTQ方法。
原文摘要 · Abstract (English)
Diffusion models have shown remarkable performance in image synthesis by progressively estimating a smooth transition from a Gaussian distribution of noise to a real image. Unfortunately, their practical deployment is limited by slow inference speed, high memory usage, and the computational demands of the noise estimation process. Post-training quantization (PTQ) emerges as a promising solution to accelerate sampling and reduce memory overhead for diffusion models. Existing PTQ methods for diffusion models typically apply uniform weights to calibration samples across timesteps, which is sub-optimal since data at different timesteps may contribute differently to the diffusion process. Additionally, due to varying activation distributions and gradients across timesteps, a uniform quantization approach is sub-optimal. Each timestep requires a different gradient direction for optimal quantization, and treating them equally can lead to conflicting gradients that degrade performance. In this paper, we propose a novel PTQ method that addresses these challenges by assigning appropriate weights to calibration samples. Specifically, our approach learns to assign optimal weights to calibration samples to align the quantized model's gradients across timesteps, facilitating the quantization process. Extensive experiments on CIFAR-10, LSUN-Bedrooms, and ImageNet demonstrate the superiority of our method compared to other PTQ methods for diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。