用扩散模型选关键图像块,一次完成高效数据蒸馏。
Towards Consistent and Efficient Dataset Distillation via Diffusion-Driven Selection
- 用扩散模型预测噪声,筛选原图中最具区分性的图像块。
- 在ImageNet-1K上仅用10%数据即达95.3%准确率,优于现有方法。
- 无需多步迭代,适合大规模数据集和复杂模型的快速蒸馏。
数据蒸馏通过优化紧凑数据集来降低内存与计算开销,实现与原始全量数据相当的性能。然而,在大规模数据集(如ImageNet-1K)和复杂网络(如ResNet-101)下,巨大的优化空间限制了蒸馏效果,制约了实际应用。现有方法利用预训练扩散模型直接生成信息量高的图像,绕过像素级优化,取得良好效果。但此类方法常因预训练扩散先验与目标数据分布差异,以及需在不同设置下多次蒸馏而受限。为此,我们提出一种新框架,与现有基于扩散的方法正交:不用于生成,而是利用扩散先验进行图像块选择。方法基于输入图像及可选文本提示(含或不含标签信息),预测扩散模型中的噪声,并计算每张图像块的损失。根据损失差异识别原图中具有代表性的区域,并结合类内聚类与排序,强化多样性约束。该简化流程支持单步蒸馏。大量实验表明,本方法在多种指标与设置下均持续优于当前最优方法。
原文摘要 · Abstract (English)
Dataset distillation provides an effective approach to reduce memory and computational costs by optimizing a compact dataset that achieves performance comparable to the full original. However, for large-scale datasets and complex deep networks (e.g., ImageNet-1K with ResNet-101), the vast optimization space hinders distillation effectiveness, limiting practical applications. Recent methods leverage pre-trained diffusion models to directly generate informative images, thereby bypassing pixel-level optimization and achieving promising results. Nonetheless, these approaches often suffer from distribution shifts between the pre-trained diffusion prior and target datasets, as well as the need for multiple distillation steps under varying settings. To overcome these challenges, we propose a novel framework that is orthogonal to existing diffusion-based distillation techniques by utilizing the diffusion prior for patch selection rather than generation. Our method predicts noise from the diffusion model conditioned on input images and optional text prompts (with or without label information), and computes the associated loss for each image-patch pair. Based on the loss differences, we identify distinctive regions within the original images. Furthermore, we apply intra-class clustering and ranking on the selected patches to enforce diversity constraints. This streamlined pipeline enables a one-step distillation process. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art methods across various metrics and settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。