通过分离内容与风格，提升数据压缩的性能和多样性。
Decomposed Distribution Matching in Dataset Condensation
- 将数据分布拆分为内容与风格，分别优化匹配。
- 在多个数据集上最高提升5.5%准确率，尤其在持续学习中表现突出。
- 方法简单有效，适合追求高精度数据压缩的研究者。
数据压缩（DC）旨在通过合成小规模数据集来减少深度神经网络的训练开销，使其效果接近原始大规模数据集。传统方法依赖昂贵的双层优化，难以实用；近期研究将其转化为分布匹配问题以提高效率，但牺牲了性能。本文通过分解数据分布为内容与风格，发现两大缺陷：原始数据与压缩数据间存在风格差异，且压缩数据类内多样性不足。为此，提出一种简单有效的方法：利用特征图的统计矩作为风格指标，匹配风格信息；并通过最大化每类内部的KL散度来增强类内多样性。实验在多种不同规模和分辨率的数据集上验证了该方法的有效性，在CIFAR10、CIFAR100、TinyImageNet、ImageNet-1K、ImageWoof、ImageNette上分别提升4.1%、4.2%、4.3%、2.0%、3.3%、2.5%，持续学习任务中更达5.5%提升。
原文摘要 · Abstract (English)
Dataset Condensation (DC) aims to reduce deep neural networks training efforts by synthesizing a small dataset such that it will be as effective as the original large dataset. Conventionally, DC relies on a costly bi-level optimization which prohibits its practicality. Recent research formulates DC as a distribution matching problem which circumvents the costly bi-level optimization. However, this efficiency sacrifices the DC performance. To investigate this performance degradation, we decomposed the dataset distribution into content and style. Our observations indicate two major shortcomings of: 1) style discrepancy between original and condensed data, and 2) limited intra-class diversity of condensed dataset. We present a simple yet effective method to match the style information between original and condensed data, employing statistical moments of feature maps as well-established style indicators. Moreover, we enhance the intra-class diversity by maximizing the Kullback-Leibler divergence within each synthetic class, i.e., content. We demonstrate the efficacy of our method through experiments on diverse datasets of varying size and resolution, achieving improvements of up to 4.1% on CIFAR10, 4.2% on CIFAR100, 4.3% on TinyImageNet, 2.0% on ImageNet-1K, 3.3% on ImageWoof, 2.5% on ImageNette, and 5.5% in continual learning accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。