提出新框架统一数据剪枝与蒸馏，提升大规模数据压缩效率。
Unifying Dataset Pruning and Distillation for Efficient Large-scale Compression
- 通过筛选易样本、有效组合与受限增强，不依赖软标签提升图像质量。
- 小数据集下剪枝性能优于蒸馏，且新方法在存储效率上显著更优。
- 适合需要高效压缩海量数据的模型训练场景，如边缘设备部署。
数据剪枝（DP）和数据蒸馏（DD）本质不同：前者选择原始图像子集，后者生成合成图像。近期研究表明，DD对原始图像的依赖日益增强，暗示两者趋于融合。为探究这一趋势，我们提出了统一的数据压缩（DC）基准。该基准揭示软标签蒸馏存在权衡：软标签虽提供信息价值，但可能削弱蒸馏必要性，因蒸馏图像未必优于随机子集。此外，在当前阶段，小规模数据集下剪枝表现优于蒸馏。基于此，我们探索硬标签-DC作为互补方案，强调图像质量并实现显著存储效率。我们的PCA（Prune, Combine, and Augment）是首个不依赖软标签的框架，通过三步实现：(1) 'P' 基于剪枝指标选取易样本；(2) 'C' 高效组合这些样本；(3) 'A' 在训练中施加受限图像增强。代码已开源。
原文摘要 · Abstract (English)
Dataset pruning (DP) and dataset distillation (DD) fundamentally differ in their outputs: DP selects original image subsets, while DD generates synthetic images. Recently, DD's increasing reliance on original images suggests a convergence of the two directions. To investigate this convergence trend, we propose a unified dataset compression (DC) benchmark. This benchmark reveals an interesting trade-off for soft-label-DD: while soft labels provide valuable information, they can make the distillation process less essential, as distilled images may not always outperform random subsets. In addition, the benchmark reveals that in current stages, dataset pruning outperforms dataset distillation at small dataset sizes. Given these observations, we explore hard-label-DC as a complementary approach that emphasizes image quality while offering substantial storage efficiency. Our PCA (Prune, Combine, and Augment) is the first framework that does not rely on soft labels but instead focuses on image quality. (1) "P'' means selecting easy samples based on dataset pruning metrics, (2) "C'' indicates combining these samples effectively, and (3) "A'' is to apply constrained image augmentation during training. Our code is available at https://github.com/ArmandXiao/Unifying-Dataset-Pruning-and-Distillation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。