用扩散模型生成高代表性的精简数据集，提升小样本训练效果
Taming Diffusion for Dataset Distillation with High Representativeness
- 通过DDIM反演将数据映射到高正态分布空间，保持结构一致性
- 新采样策略使生成数据更贴近理想分布，准确率显著提升
- 适用于需要压缩数据量又不牺牲性能的模型训练场景
当前深度学习模型依赖大规模数据集，推动了数据集蒸馏技术的发展，旨在生成紧凑且高效的替代数据集。由于扩散模型具备强大的图像生成能力，已被用于生成蒸馏数据。本文系统分析了现有基于扩散模型的数据集蒸馏方法中存在的问题，包括分布匹配不准、随机噪声导致分布偏移以及采样过程分离等。为此，我们提出D^3HR框架，通过DDIM反演将完整数据集的潜在表示从低正态性空间映射到高正态性高斯空间，保留信息并确保结构一致性，从而生成具有高代表性的蒸馏数据潜在表示。此外，设计了一种高效采样方案，使代表性潜在变量更好地对齐高正态性高斯分布。大量实验表明，D^3HR在不同模型架构下均优于现有最先进基线方法，在数据集蒸馏任务中实现更高准确率。代码已开源：https://github.com/lin-zhao-resoLve/D3HR。
原文摘要 · Abstract (English)
Recent deep learning models demand larger datasets, driving the need for dataset distillation to create compact, cost-efficient datasets while maintaining performance. Due to the powerful image generation capability of diffusion, it has been introduced to this field for generating distilled images. In this paper, we systematically investigate issues present in current diffusion-based dataset distillation methods, including inaccurate distribution matching, distribution deviation with random noise, and separate sampling. Building on this, we propose D^3HR, a novel diffusion-based framework to generate distilled datasets with high representativeness. Specifically, we adopt DDIM inversion to map the latents of the full dataset from a low-normality latent domain to a high-normality Gaussian domain, preserving information and ensuring structural consistency to generate representative latents for the distilled dataset. Furthermore, we propose an efficient sampling scheme to better align the representative latents with the high-normality Gaussian distribution. Our comprehensive experiments demonstrate that D^3HR can achieve higher accuracy across different model architectures compared with state-of-the-art baselines in dataset distillation. Source code: https://github.com/lin-zhao-resoLve/D3HR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。