用扩散模型生成数据，实现无需真实训练数据的知识蒸馏。
Data-free Knowledge Distillation with Diffusion Models
- 用教师模型指导扩散模型生成贴近真实数据分布的图像。
- 提出潜空间CutMix增强，提升生成图像多样性且降低计算开销。
- 在多个数据集上超越现有方法，适合隐私敏感场景使用。
最近，无数据知识蒸馏（Data-Free Knowledge Distillation, DFKD）受到关注，可在不访问训练数据的情况下将知识从教师网络迁移至学生网络。尽管扩散模型擅长在多个领域生成高保真、逼真的图像，但现有方法难以直接应用于DFKD。为此，本文提出一种基于扩散模型的新方法DiffDFKD。具体而言，DiffDFKD在两个关键环节进行优化：首先，利用教师模型中的有用信息引导预训练扩散模型的数据生成，生成符合训练数据分布的图像，有效弥合领域差异；其次，为减轻计算负担，引入潜空间CutMix增强技术，提升扩散模型生成图像的多样性，同时保留对知识迁移至关重要的特征。大量实验验证了DiffDFKD的有效性，在多个基准上达到当前最优性能。代码已开源：https://github.com/xhqi0109/DiffDFKD。
原文摘要 · Abstract (English)
Recently Data-Free Knowledge Distillation (DFKD) has garnered attention and can transfer knowledge from a teacher neural network to a student neural network without requiring any access to training data. Although diffusion models are adept at synthesizing high-fidelity photorealistic images across various domains, existing methods cannot be easiliy implemented to DFKD. To bridge that gap, this paper proposes a novel approach based on diffusion models, DiffDFKD. Specifically, DiffDFKD involves targeted optimizations in two key areas. Firstly, DiffDFKD utilizes valuable information from teacher models to guide the pre-trained diffusion models' data synthesis, generating datasets that mirror the training data distribution and effectively bridge domain gaps. Secondly, to reduce computational burdens, DiffDFKD introduces Latent CutMix Augmentation, an efficient technique, to enhance the diversity of diffusion model-generated images for DFKD while preserving key attributes for effective knowledge transfer. Extensive experiments validate the efficacy of DiffDFKD, yielding state-of-the-art results exceeding existing DFKD approaches. We release our code at https://github.com/xhqi0109/DiffDFKD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。