arXiv:2412.05926cs.CV2024-12NeurIPS被引 11

提出BiDM方法,实现扩散模型权重和激活的全1比特量化,大幅压缩存储与计算开销。

BiDM: Pushing the Limit of Quantization for Diffusion Models

  • 从时间与空间双角度设计结构,解决1比特量化下特征相关性与匹配难题。
  • 在LSUN-Bedrooms数据集上达到FID 22.74,显著优于现有方法(59.44)。
  • 适合资源受限场景下的高效扩散模型部署,如移动端或边缘设备。

扩散模型(DMs)因生成质量优异而广泛应用,但其高计算成本与庞大参数限制了在资源受限场景中的应用。量化可通过降低位宽实现存储压缩与推理加速,但最极端的1比特二值化会导致生成性能严重退化甚至崩溃。本文提出BiDM方法,首次实现对扩散模型权重和激活的完全1比特二值化。从时序角度,引入时间友好的二值化结构(TBS),通过可学习激活二值化器与跨时间步特征连接,缓解扩散模型中高度依赖时间步的激活特征问题;从空间角度,提出空间分块蒸馏(SPD),聚焦图像生成任务的空间局部性与噪声估计网络特性,提升二值特征的蒸馏匹配能力。在LDM-4模型与LSUN-Bedrooms 256×256数据集上,W1A1 BiDM取得22.74的FID,远超当前最佳通用二值化方法(59.44),且生成样本有效,实现最高28.0倍存储节省与52.7倍操作数节省。代码已开源。

原文摘要 · Abstract (English)

Diffusion models (DMs) have been significantly developed and widely used in various applications due to their excellent generative qualities. However, the expensive computation and massive parameters of DMs hinder their practical use in resource-constrained scenarios. As one of the effective compression approaches, quantization allows DMs to achieve storage saving and inference acceleration by reducing bit-width while maintaining generation performance. However, as the most extreme quantization form, 1-bit binarization causes the generation performance of DMs to face severe degradation or even collapse. This paper proposes a novel method, namely BiDM, for fully binarizing weights and activations of DMs, pushing quantization to the 1-bit limit. From a temporal perspective, we introduce the Timestep-friendly Binary Structure (TBS), which uses learnable activation binarizers and cross-timestep feature connections to address the highly timestep-correlated activation features of DMs. From a spatial perspective, we propose Space Patched Distillation (SPD) to address the difficulty of matching binary features during distillation, focusing on the spatial locality of image generation tasks and noise estimation networks. As the first work to fully binarize DMs, the W1A1 BiDM on the LDM-4 model for LSUN-Bedrooms 256$\times$256 achieves a remarkable FID of 22.74, significantly outperforming the current state-of-the-art general binarization methods with an FID of 59.44 and invalid generative samples, and achieves up to excellent 28.0 times storage and 52.7 times OPs savings. The code is available at https://github.com/Xingyu-Zheng/BiDM .

扩散模型1比特量化模型压缩高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。