将真实数据知识蒸馏到生成模型参数中，实现高效、可扩展的少样本学习。
Data-to-Model Distillation: Data-Efficient Learning Framework
- 用预训练生成模型参数存储数据知识，替代传统像素级蒸馏。
- 在15个不同分辨率数据集上表现优于现有方法，支持128x128 ImageNet-1K。
- 可适配不同压缩比和深度网络结构，适合神经架构搜索等下游任务。
数据蒸馏旨在将大规模真实数据集的知识浓缩为少量但信息丰富的合成数据，使在该数据上训练的模型性能接近于在全量数据上训练的结果。尽管已有进展，现有方法在计算效率、高分辨率复杂数据集的可扩展性以及对深层架构的泛化能力方面仍存在不足。这些方法通常在改变蒸馏比例时需重新训练，因知识嵌入在原始像素中。本文提出一种新框架——数据到模型蒸馏（D2M），通过对齐真实图像与生成图像提取的丰富特征表示，将真实数据的知识蒸馏到预训练生成模型的可学习参数中。由此获得的生成模型可针对不同蒸馏比例和深层架构生成高质量训练图像。在15个不同分辨率的数据集上进行的大量实验表明，D2M在性能、重蒸馏效率和跨架构泛化性方面均表现优异。该方法有效扩展至128x128分辨率的ImageNet-1K数据集，并验证了其在神经架构搜索等下游应用中的实际价值。
原文摘要 · Abstract (English)
Dataset distillation aims to distill the knowledge of a large-scale real dataset into small yet informative synthetic data such that a model trained on it performs as well as a model trained on the full dataset. Despite recent progress, existing dataset distillation methods often struggle with computational efficiency, scalability to complex high-resolution datasets, and generalizability to deep architectures. These approaches typically require retraining when the distillation ratio changes, as knowledge is embedded in raw pixels. In this paper, we propose a novel framework called Data-to-Model Distillation (D2M) to distill the real dataset's knowledge into the learnable parameters of a pre-trained generative model by aligning rich representations extracted from real and generated images. The learned generative model can then produce informative training images for different distillation ratios and deep architectures. Extensive experiments on 15 datasets of varying resolutions show D2M's superior performance, re-distillation efficiency, and cross-architecture generalizability. Our method effectively scales up to high-resolution 128x128 ImageNet-1K. Furthermore, we verify D2M's practical benefits for downstream applications in neural architecture search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。