用单哈达玛变换提升低比特扩散模型的量化精度，减少推理时的异常值影响。
HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models
- 引入单哈达玛变换处理激活矩阵，抑制量化过程中的异常值。
- 在ImageNet 256x256上，W4A3比当前最优方法提升Inception Score 467.73%。
- 兼容整数卷积，避免权重异常值放大，适合部署于资源受限设备。
扩散模型在图像生成领域已展现出广泛应用，但其高计算与内存开销给部署带来挑战。模型量化成为降低存储和加速推理的可行方案。然而，现有扩散模型量化方法在低比特场景下难以缓解激活矩阵中的异常值，导致性能显著下降。为此，我们提出HQ-DM，一种基于单哈达玛变换的量化感知训练框架。该方法有效降低激活异常值，同时保持量化后模型性能。相比传统双哈达玛变换，本方案可无缝支持整数卷积运算，并防止权重异常值放大。在ImageNet 256x256数据集上使用LDM-4模型进行条件生成时，我们的W4A4和W4A3量化方案相较现有最先进方法，分别提升Inception Score 12.8%和467.73%。
原文摘要 · Abstract (English)
Diffusion models have demonstrated significant applications in the field of image generation. However, their high computational and memory costs pose challenges for deployment. Model quantization has emerged as a promising solution to reduce storage overhead and accelerate inference. Nevertheless, existing quantization methods for diffusion models struggle to mitigate outliers in activation matrices during inference, leading to substantial performance degradation under low-bit quantization scenarios. To address this, we propose HQ-DM, a novel Quantization-Aware Training framework that applies Single Hadamard Transformation to activation matrices. This approach effectively reduces activation outliers while preserving model performance under quantization. Compared to traditional Double Hadamard Transformation, our proposed scheme offers distinct advantages by seamlessly supporting INT convolution operations while preventing the amplification of weight outliers. For conditional generation on the ImageNet 256x256 dataset using the LDM-4 model, our W4A4 and W4A3 quantization schemes improve the Inception Score by 12.8% and 467.73%, respectively, over the existing state-of-the-art method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。