量化与缓存结合加速图像生成,解决性能崩溃问题。
Q&C: When Quantization Meets Cache in Efficient Image Generation
- 动态聚类优化校准数据选择,适配不同扩散步骤。
- 12.7倍加速同时保持高质量图像生成能力。
- 适合追求高效扩散模型推理的开发者与研究者。
量化和缓存机制通常单独用于高效扩散Transformer(DiTs)的加速,各自展现出显著潜力。然而,两者结合对高效生成的促进作用尚未深入探索。实证研究发现,将二者结合并非简单叠加,存在两大挑战导致严重性能下降:(i) 缓存操作显著削弱了后训练量化(PTQ)中校准数据集的样本有效性;(ii) 两种机制结合引入更严重的采样分布偏差,加剧生成过程中的误差累积。为此,本文提出一种混合加速方法,通过应对上述挑战,在保持优异生成能力的同时进一步提升效率。具体地,设计了时间感知并行聚类(TAP),动态提升不同扩散步骤下校准阶段的样本选择效能;提出方差补偿(VC)策略,通过自适应校正因子生成缓解采样偏差。大量实验表明,本方法使DiTs加速达12.7倍,同时保持竞争性生成质量。代码将发布于https://github.com/xinding-sys/Quant-Cache。
原文摘要 · Abstract (English)
Quantization and cache mechanisms are typically applied individually for efficient Diffusion Transformers (DiTs), each demonstrating notable potential for acceleration. However, the promoting effect of combining the two mechanisms on efficient generation remains under-explored. Through empirical investigation, we find that the combination of quantization and cache mechanisms for DiT is not straightforward, and two key challenges lead to severe catastrophic performance degradation: (i) the sample efficacy of calibration datasets in post-training quantization (PTQ) is significantly eliminated by cache operation; (ii) the combination of the above mechanisms introduces more severe exposure bias within sampling distribution, resulting in amplified error accumulation in the image generation process. In this work, we take advantage of these two acceleration mechanisms and propose a hybrid acceleration method by tackling the above challenges, aiming to further improve the efficiency of DiTs while maintaining excellent generation capability. Concretely, a temporal-aware parallel clustering (TAP) is designed to dynamically improve the sample selection efficacy for the calibration within PTQ for different diffusion steps. A variance compensation (VC) strategy is derived to correct the sampling distribution. It mitigates exposure bias through an adaptive correction factor generation. Extensive experiments have shown that our method has accelerated DiTs by 12.7x while preserving competitive generation capability. The code will be available at https://github.com/xinding-sys/Quant-Cache.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。