针对扩散模型量化中权重与激活分离重建问题，提出时间感知量化方法。
TaQ-DiT: Time-aware Quantization for Diffusion Transformers
- 联合重构权重与激活，解决传统方法的非收敛问题。
- 发现Post-GELU层在不同去噪步骤中对量化敏感，提出时间变异性感知变换。
- 在W4A8量化下性能显著优于现有方法，适合部署高精度生成模型。
基于Transformer的扩散模型（称为DiTs）在图像和视频生成任务中达到了顶尖性能。然而，其庞大的模型规模和缓慢的推理速度限制了实际应用，亟需如量化等模型压缩技术。遗憾的是，现有的DiT量化方法忽视了（1）重构过程的影响，以及（2）不同层间量化敏感性的差异，从而制约了性能提升。为此，我们提出面向DiTs的时间感知量化方法（TaQ-DiT）。具体而言，（1）我们发现，在量化过程中分别重构权重与激活会导致非收敛问题，因此引入联合重构方法予以解决；（2）我们发现Post-GELU激活在不同去噪步骤中具有显著的变异性、极端不对称性及内部波动，因而对量化极为敏感。为应对该问题，我们提出时间变异性感知变换以实现更有效的量化。实验结果表明，当将DiTs的权重量化至4比特、激活量化至8比特（即W4A8）时，本方法显著超越此前的量化方法。
原文摘要 · Abstract (English)
Transformer-based diffusion models, dubbed Diffusion Transformers (DiTs), have achieved state-of-the-art performance in image and video generation tasks. However, their large model size and slow inference speed limit their practical applications, calling for model compression methods such as quantization. Unfortunately, existing DiT quantization methods overlook (1) the impact of reconstruction and (2) the varying quantization sensitivities across different layers, which hinder their achievable performance. To tackle these issues, we propose innovative time-aware quantization for DiTs (TaQ-DiT). Specifically, (1) we observe a non-convergence issue when reconstructing weights and activations separately during quantization and introduce a joint reconstruction method to resolve this problem. (2) We discover that Post-GELU activations are particularly sensitive to quantization due to their significant variability across different denoising steps as well as extreme asymmetries and variations within each step. To address this, we propose time-variance-aware transformations to facilitate more effective quantization. Experimental results show that when quantizing DiTs' weights to 4-bit and activations to 8-bit (W4A8), our method significantly surpasses previous quantization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。