通过激进量化与时间稀疏性加速扩散模型生成速度。
SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity
- 采用4比特量化与通道-时间动态稀疏性设计。
- 相比传统加速器实现6.91倍提速与51.5%能效降低。
- 适合追求高效率图像生成的部署场景。
扩散模型在图像生成任务中广受欢迎，但高质量生成仍因需多步推理而缓慢。本文提出激进量化权重与激活的同时，大幅促进激活稀疏性，并发现不同通道的稀疏模式随时间步变化。为此，设计了异构混合精度稠密-稀疏架构、通道优先地址映射及时间感知稀疏检测器以高效处理稀疏模式。4比特量化方法在生成质量上优于现有4比特方案。定制加速器相较传统稠密加速器实现6.91倍速度提升和51.5%能耗降低。
原文摘要 · Abstract (English)
Diffusion models have gained significant popularity in image generation tasks. However, generating high-quality content remains notably slow because it requires running model inference over many time steps. To accelerate these models, we propose to aggressively quantize both weights and activations, while simultaneously promoting significant activation sparsity. We further observe that the stated sparsity pattern varies among different channels and evolves across time steps. To support this quantization and sparsity scheme, we present a novel diffusion model accelerator featuring a heterogeneous mixed-precision dense-sparse architecture, channel-last address mapping, and a time-step-aware sparsity detector for efficient handling of the sparsity pattern. Our 4-bit quantization technique demonstrates superior generation quality compared to existing 4-bit methods. Our custom accelerator achieves 6.91x speed-up and 51.5% energy reduction compared to traditional dense accelerators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。