用扩散模型生成多样数据，提升无数据知识蒸馏效果
Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation
- 用扩散模型对合成数据进行自监督增强，提升多样性
- 在三个数据集上优于现有最先进方法，性能显著
- 适合做模型压缩且无原始数据的场景
无数据知识蒸馏（DFKD）是模型压缩领域的重要技术，大幅降低对原始训练数据的依赖。然而，传统DFKD方法生成的合成数据存在多样性不足、与真实数据分布差异大的问题。本文提出一种基于多样化扩散增强（DDA）的新方法：将常规数据合成流程升级为先合成后利用扩散模型进行自监督增强的复合过程，生成分布相似但具有可控差异的数据样本；同时引入基于余弦相似度的图像筛选机制，抑制嵌入空间过度偏离，保障蒸馏精度。在CIFAR-10、CIFAR-100和Tiny-ImageNet上的大量实验表明，该方法在多种师生网络配置下均优于当前最先进的DFKD方法。代码将在https://github.com/SLGSP/DDA发布。
原文摘要 · Abstract (English)
Data-free knowledge distillation (DFKD) has emerged as a pivotal technique in the domain of model compression, substantially reducing the dependency on the original training data. Nonetheless, conventional DFKD methods that employ synthesized training data are prone to the limitations of inadequate diversity and discrepancies in distribution between the synthesized and original datasets. To address these challenges, this paper introduces an innovative approach to DFKD through diverse diffusion augmentation (DDA). Specifically, we revise the paradigm of common data synthesis in DFKD to a composite process through leveraging diffusion models subsequent to data synthesis for self-supervised augmentation, which generates a spectrum of data samples with similar distributions while retaining controlled variations. Furthermore, to mitigate excessive deviation in the embedding space, we introduce an image filtering technique grounded in cosine similarity to maintain fidelity during the knowledge distillation process. Comprehensive experiments conducted on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets showcase the superior performance of our method across various teacher-student network configurations, outperforming the contemporary state-of-the-art DFKD methods. Code will be available at:https://github.com/SLGSP/DDA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。