扩散模型可先预训练,再轻量微调实现一步生成。
Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
- 用生成对抗网络微调预训练扩散模型,避免传统蒸馏的性能下降。
- 仅用0.2M图像微调即达强性能,5M图像接近最先进水平。
- 揭示扩散训练本质是生成式预训练,适合追求高效生成的应用。
扩散蒸馏虽能降低采样成本,但常需大量训练且学生模型性能下降。近期研究发现引入GAN目标可缓解此问题,但机制不明。本文指出蒸馏失效主因:教师与学生模型步长和参数量不匹配,导致收敛至不同局部最优。我们进一步证明,仅使用GAN目标即可克服此问题,并足以将扩散模型转化为高效的一步生成器。基于此,我们提出扩散训练本质上是一种生成式预训练,可通过轻量级GAN微调解锁生成能力。实验显示,冻结85%参数的预训练模型仅用0.2M图像微调即表现优异,5M图像微调达到近似最先进结果。我们还通过频域分析解释了一步生成能力的来源。本工作为扩散模型提供了新视角,强调其作为生成预训练的基础价值。
原文摘要 · Abstract (English)
Diffusion distillation is a widely used technique to reduce the sampling cost of diffusion models, yet it often requires extensive training, and the student performance tends to be degraded. Recent studies show that incorporating a GAN objective may alleviate these issues, yet the underlying mechanism remains unclear. In this work, we first identify a key limitation of distillation: mismatched step sizes and parameter numbers between the teacher and the student model lead them to converge to different local minima, rendering direct imitation suboptimal. We further demonstrate that a standalone GAN objective, without relying a distillation loss, overcomes this limitation and is sufficient to convert diffusion models into efficient one-step generators. Based on this finding, we propose that diffusion training may be viewed as a form of generative pre-training, equipping models with capabilities that can be unlocked through lightweight GAN fine-tuning. Supporting this view, we create a one-step generation model by fine-tuning a pre-trained model with 85% of parameters frozen, achieving strong performance with only 0.2M images and near-SOTA results with 5M images. We further present a frequency-domain analysis that may explain the one-step generative capability gained in diffusion training. Overall, our work provides a new perspective for diffusion training, highlighting its role as a powerful generative pre-training process, which can be the basis for building efficient one-step generation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。