arXiv:2510.17421cs.LG2025-10被引 2

用扩散模型的特征相似性提升数据蒸馏的代表性,无需重训练。

Diffusion Models as Dataset Distillation Priors

  • 基于特征空间核相似性定义代表性的先验,指导反向扩散过程。
  • 在ImageNet-1K等数据集上生成的蒸馏数据质量超越现有方法。
  • 无需重训练,适合追求高保真与跨架构泛化的研究者。

数据蒸馏旨在从大规模数据集中合成紧凑且信息丰富的数据集。该领域面临的核心挑战是同时实现多样性、泛化能力和代表性。尽管近年生成式数据蒸馏方法采用强大的扩散模型作为基础模型,但其内在的代表性先验被忽视,导致这些方法常需引入外部约束以提升数据质量。为此,我们提出扩散模型作为先验(DAP),通过默克尔核量化合成数据与真实数据在特征空间中的相似性,形式化代表性的先验。随后将此先验作为引导,用于调节反向扩散过程,从而在不进行任何重训练的情况下增强蒸馏样本的代表性。在ImageNet-1K及其子集等大规模数据集上的大量实验表明,DAP在生成高保真数据集的同时,实现了更优的跨架构泛化能力。本工作不仅建立了扩散先验与数据蒸馏目标之间的理论联系,还提供了一个实际、无需训练的框架,显著提升蒸馏数据质量。

原文摘要 · Abstract (English)

Dataset distillation aims to synthesize compact yet informative datasets from large ones. A significant challenge in this field is achieving a trifecta of diversity, generalization, and representativeness in a single distilled dataset. Although recent generative dataset distillation methods adopt powerful diffusion models as their foundation models, the inherent representativeness prior in diffusion models is overlooked. Consequently, these approaches often necessitate the integration of external constraints to enhance data quality. To address this, we propose Diffusion As Priors (DAP), which formalizes representativeness by quantifying the similarity between synthetic and real data in feature space using a Mercer kernel. We then introduce this prior as guidance to steer the reverse diffusion process, enhancing the representativeness of distilled samples without any retraining. Extensive experiments on large-scale datasets, such as ImageNet-1K and its subsets, demonstrate that DAP outperforms state-of-the-art methods in generating high-fidelity datasets while achieving superior cross-architecture generalization. Our work not only establishes a theoretical connection between diffusion priors and the objectives of dataset distillation but also provides a practical, training-free framework for improving the quality of the distilled dataset.

数据蒸馏扩散模型无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。