不微调模型,通过降噪步数与推理成本的权衡,实现扩散模型高效部署。
Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment
- 提出PostDiff框架,训练后压缩模型冗余,降低推理开销。
- 减少每步推理成本比减少降噪步数更有效,兼顾生成质量与效率。
- 适合资源受限设备部署,尤其对生成图像质量要求高的场景。
扩散模型在生成任务中表现卓越,但其高计算需求限制了在资源受限平台的部署。本文探讨在不微调模型的前提下,减少降噪步数或降低每步推理成本何者更优。研究表明,减少降噪步数会增加各步分布差异,使模型对压缩更敏感;而保留更多步数可缩小差异、保留冗余,更利于训练后压缩。为此,我们提出PostDiff框架,通过输入级混合分辨率去噪和模块级混合缓存策略,在训练后阶段减少冗余。前者在早期去噪步骤降低生成分辨率以增强低频成分,提升最终生成质量;后者复用跨步骤计算。大量实验表明:(1)PostDiff显著改善主流扩散模型的保真度-效率权衡;(2)为提升效率且维持良好生成质量,降低每步推理成本通常优于减少降噪步数。代码已开源。
原文摘要 · Abstract (English)
Diffusion models have shown remarkable success across generative tasks, yet their high computational demands challenge deployment on resource-limited platforms. This paper investigates a critical question for compute-optimal diffusion model deployment: Under a post-training setting without fine-tuning, is it more effective to reduce the number of denoising steps or to use a cheaper per-step inference? Intuitively, reducing the number of denoising steps increases the variability of the distributions across steps, making the model more sensitive to compression. In contrast, keeping more denoising steps makes the differences smaller, preserving redundancy, and making post-training compression more feasible. To systematically examine this, we propose PostDiff, a training-free framework for accelerating pre-trained diffusion models by reducing redundancy at both the input level and module level in a post-training manner. At the input level, we propose a mixed-resolution denoising scheme based on the insight that reducing generation resolution in early denoising steps can enhance low-frequency components and improve final generation fidelity. At the module level, we employ a hybrid module caching strategy to reuse computations across denoising steps. Extensive experiments and ablation studies demonstrate that (1) PostDiff can significantly improve the fidelity-efficiency trade-off of state-of-the-art diffusion models, and (2) to boost efficiency while maintaining decent generation fidelity, reducing per-step inference cost is often more effective than reducing the number of denoising steps. Our code is available at https://github.com/GATECH-EIC/PostDiff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。