提出DilateQuant,让扩散模型量化更准更快。
DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation
- 用权重膨胀技术控制激活范围,稳定量化误差。
- 低比特下精度比现有方法高1.5~3.2个点,训练效率提升2倍。
- 适合需要高效部署扩散模型的开发者和研究者。
模型量化是加速和压缩扩散模型的有效方法。然而,后训练量化(PTQ)在低比特情况下会严重失效,因此量化感知训练(QAT)至关重要。由于扩散模型中激活值范围广且随时间变化,现有QAT方法效率低下。等效缩放虽能减小激活范围,但无法降低整体量化误差,且显著破坏原始权重分布,导致权重初始化不良,QAT训练难以收敛。本文提出一种新的扩散模型QAT框架DilateQuant。核心是权重膨胀(WD),通过等效缩放将未饱和的通道内权重最大范围地拉伸至受限区间,既缩小激活范围又保持原始权重分布,从而持续降低量化误差并确保模型收敛。为进一步提升准确率与效率,设计了时序并行量化器(TPQ)应对时变激活,以及块级知识蒸馏(BKD)以减少训练资源消耗。大量实验表明,DilateQuant在准确率和效率上均显著优于现有方法。代码已开源。
原文摘要 · Abstract (English)
Model quantization is a promising method for accelerating and compressing diffusion models. Nevertheless, since post-training quantization (PTQ) fails catastrophically at low-bit cases, quantization-aware training (QAT) is essential. Unfortunately, the wide range and time-varying activations in diffusion models sharply increase the complexity of quantization, making existing QAT methods inefficient. Equivalent scaling can effectively reduce activation range, but previous methods remain the overall quantization error unchanged. More critically, these methods significantly disrupt the original weight distribution, resulting in poor weight initialization and challenging convergence during QAT training. In this paper, we propose a novel QAT framework for diffusion models, called DilateQuant. Specifically, we propose Weight Dilation (WD) that maximally dilates the unsaturated in-channel weights to a constrained range through equivalent scaling. WD decreases the activation range while preserving the original weight range, which steadily reduces the quantization error and ensures model convergence. To further enhance accuracy and efficiency, we design a Temporal Parallel Quantizer (TPQ) to address the time-varying activations and introduce a Block-wise Knowledge Distillation (BKD) to reduce resource consumption in training. Extensive experiments demonstrate that DilateQuant significantly outperforms existing methods in terms of accuracy and efficiency. Code is available at http://github.com/BienLuky/DilateQuant .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。