用对抗方式改进扩散模型压缩,避免生成模式坍塌。
Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis
- 引入对抗式分布匹配,用扩散判别器对齐真实与虚假得分估计器的潜在分布。
- 在单步蒸馏上超越DMD2,SDXL模型性能提升且显存消耗更低。
- 适用于需要高效图像视频生成的场景,尤其适合资源受限部署。
分布匹配蒸馏(DMD)是一种有前景的分数蒸馏技术,可将预训练的教师扩散模型压缩为高效的单步或多步学生生成器。然而,其依赖反向KL散度最小化可能在某些应用中引发模式坍塌(或模式寻找)。为克服这一固有缺陷,我们提出对抗式分布匹配(ADM),一种新框架,利用基于扩散的判别器以对抗方式对齐真实与虚假分数估计器之间的潜在预测分布,实现分数蒸馏。在极具挑战性的单步蒸馏场景中,我们进一步通过混合判别器在潜在空间和像素空间进行对抗蒸馏,优化预训练生成器。不同于DMD2预训练中使用的均方误差,本方法在教师模型收集的ODE对上引入分布损失,从而为后续分数蒸馏微调阶段提供更优初始化。将对抗蒸馏预训练与ADM微调整合为统一流程DMDX,所提方法在SDXL上的单步性能优于DMD2,同时显存占用更低。在SD3-Medium、SD3.5-Large和CogVideoX上进行多步ADM蒸馏的附加实验,为高效图像与视频合成设立了新基准。
原文摘要 · Abstract (English)
Distribution Matching Distillation (DMD) is a promising score distillation technique that compresses pre-trained teacher diffusion models into efficient one-step or multi-step student generators. Nevertheless, its reliance on the reverse Kullback-Leibler (KL) divergence minimization potentially induces mode collapse (or mode-seeking) in certain applications. To circumvent this inherent drawback, we propose Adversarial Distribution Matching (ADM), a novel framework that leverages diffusion-based discriminators to align the latent predictions between real and fake score estimators for score distillation in an adversarial manner. In the context of extremely challenging one-step distillation, we further improve the pre-trained generator by adversarial distillation with hybrid discriminators in both latent and pixel spaces. Different from the mean squared error used in DMD2 pre-training, our method incorporates the distributional loss on ODE pairs collected from the teacher model, and thus providing a better initialization for score distillation fine-tuning in the next stage. By combining the adversarial distillation pre-training with ADM fine-tuning into a unified pipeline termed DMDX, our proposed method achieves superior one-step performance on SDXL compared to DMD2 while consuming less GPU time. Additional experiments that apply multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX set a new benchmark towards efficient image and video synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。