针对扩散模型量化中的异常值问题,提出新方法提升低比特部署性能
DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization
- 结合学习型缩放与通道级2的幂次缩放,优化权重和激活的量化分布
- 在W4A6和W4A8下显著提升图像生成质量,保持模型稳定
- 适合需要低资源部署扩散模型的研究者与工程师
扩散模型在图像生成中取得显著成功,但计算开销大,限制其在资源受限环境中的部署。现有后训练量化(PTQ)方法虽关注扩散模型的迭代特性,却常忽略异常值,导致低比特下性能下降。本文提出DMQ,融合学习型等效缩放(LES)与通道级2的幂次缩放(PTS),有效应对该问题。LES通过优化通道级缩放因子,重新分配权重与激活的量化难度,降低整体量化误差;考虑到早期去噪步骤虽量化误差小,但因误差累积对最终输出影响关键,引入自适应时间步加权机制以强化学习。针对跳接层等存在高通道间方差的结构,采用通道级PTS处理激活。为保障小校准集下PTS因子选择的可靠性,设计投票算法。大量实验表明,本方法在低比特如W4A6(4-bit权重,6-bit激活)和W4A8下显著优于现有工作,维持高质量图像生成与模型稳定性。代码已开源。
原文摘要 · Abstract (English)
Diffusion models have achieved remarkable success in image generation but come with significant computational costs, posing challenges for deployment in resource-constrained environments. Recent post-training quantization (PTQ) methods have attempted to mitigate this issue by focusing on the iterative nature of diffusion models. However, these approaches often overlook outliers, leading to degraded performance at low bit-widths. In this paper, we propose a DMQ which combines Learned Equivalent Scaling (LES) and channel-wise Power-of-Two Scaling (PTS) to effectively address these challenges. Learned Equivalent Scaling optimizes channel-wise scaling factors to redistribute quantization difficulty between weights and activations, reducing overall quantization error. Recognizing that early denoising steps, despite having small quantization errors, crucially impact the final output due to error accumulation, we incorporate an adaptive timestep weighting scheme to prioritize these critical steps during learning. Furthermore, identifying that layers such as skip connections exhibit high inter-channel variance, we introduce channel-wise Power-of-Two Scaling for activations. To ensure robust selection of PTS factors even with small calibration set, we introduce a voting algorithm that enhances reliability. Extensive experiments demonstrate that our method significantly outperforms existing works, especially at low bit-widths such as W4A6 (4-bit weight, 6-bit activation) and W4A8, maintaining high image generation quality and model stability. The code is available at https://github.com/LeeDongYeun/dmq.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。