arXiv:2409.17612cs.LGcs.CV2024-09NeurIPS被引 43

通过动态调权重提升合成数据多样性,让小数据集更高效。

Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment

  • 用动态权重调控合成过程,避免数据冗余。
  • 在CIFAR、Tiny-ImageNet等数据集上表现优于现有方法。
  • 适合需要高效数据压缩的模型训练场景。

数据成本激增推动了数据集压缩研究。数据蒸馏旨在生成可替代原始数据集的合成数据,以训练神经网络。为避免合成数据冗余,必须确保每个样本具有独特特征且彼此多样。本文对合成数据中的多样性进行了理论与实证分析,提出一种基于动态定向权重调整的新方法,通过调节合成过程最大化每条数据的代表性与多样性。该方法使每批合成数据均能反映原始数据集的大规模、多变子集特征。在CIFAR、Tiny-ImageNet和ImageNet-1K等多个数据集上的大量实验表明,该方法能以极低计算开销生成更具代表性和多样性的合成数据,性能显著优于现有方法。代码已开源:https://github.com/AngusDujw/Diversity-Driven-Synthesis。

原文摘要 · Abstract (English)

The sharp increase in data-related expenses has motivated research into condensing datasets while retaining the most informative features. Dataset distillation has thus recently come to the fore. This paradigm generates synthetic datasets that are representative enough to replace the original dataset in training a neural network. To avoid redundancy in these synthetic datasets, it is crucial that each element contains unique features and remains diverse from others during the synthesis stage. In this paper, we provide a thorough theoretical and empirical analysis of diversity within synthesized datasets. We argue that enhancing diversity can improve the parallelizable yet isolated synthesizing approach. Specifically, we introduce a novel method that employs dynamic and directed weight adjustment techniques to modulate the synthesis process, thereby maximizing the representativeness and diversity of each synthetic instance. Our method ensures that each batch of synthetic data mirrors the characteristics of a large, varying subset of the original dataset. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, demonstrate the superior performance of our method, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense. Our code is available at https://github.com/AngusDujw/Diversity-Driven-Synthesis.https://github.com/AngusDujw/Diversity-Driven-Synthesis.

数据蒸馏多样性增强合成数据高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。