arXiv:2409.03550cs.CVcs.AI2024-09CVPR被引 38

无需数据即可训练新扩散模型,用已有模型迁移生成能力。

DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture

  • 用现有扩散模型作为知识源,无须原始数据训练新模型。
  • 在不访问数据情况下,生成质量接近甚至超过全量数据训练的模型。
  • 适合资源受限或数据隐私敏感场景,支持任意架构新模型训练。

扩散模型(DMs)在图像、视频等多个领域展现出卓越的生成能力,其效果高度依赖训练数据的数量与质量。然而,主流扩散模型训练需海量数据,例如训练 Stable Diffusion 需要数十亿张图像-文本对,导致高昂的数据获取与存储成本。为缓解这一问题,本文提出一种新范式:利用现有扩散模型作为数据源,训练任意架构的新扩散模型,称为数据无关的扩散模型知识蒸馏(DKDM)。我们提出 DKDM 目标,使新模型可通过蒸馏训练而无需访问真实数据;并设计动态迭代蒸馏方法,高效提取时间域知识,实现无需长时间生成即可直接获取训练信号。实验表明,该数据无关方法不仅生成性能优异,部分情况下还超越全量数据训练模型。据我们所知,这是首个探索此场景的工作。

原文摘要 · Abstract (English)

Diffusion models (DMs) have demonstrated exceptional generative capabilities across various domains, including image, video, and so on. A key factor contributing to their effectiveness is the high quantity and quality of data used during training. However, mainstream DMs now consume increasingly large amounts of data. For example, training a Stable Diffusion model requires billions of image-text pairs. This enormous data requirement poses significant challenges for training large DMs due to high data acquisition costs and storage expenses. To alleviate this data burden, we propose a novel scenario: using existing DMs as data sources to train new DMs with any architecture. We refer to this scenario as Data-Free Knowledge Distillation for Diffusion Models (DKDM), where the generative ability of DMs is transferred to new ones in a data-free manner. To tackle this challenge, we make two main contributions. First, we introduce a DKDM objective that enables the training of new DMs via distillation, without requiring access to the data. Second, we develop a dynamic iterative distillation method that efficiently extracts time-domain knowledge from existing DMs, enabling direct retrieval of training data without the need for a prolonged generative process. To the best of our knowledge, we are the first to explore this scenario. Experimental results demonstrate that our data-free approach not only achieves competitive generative performance but also, in some instances, outperforms models trained with the entire dataset.

扩散模型知识蒸馏数据无关生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。