arXiv:2606.18209cs.LG2026-06

对比压缩数据与精选数据,发现现有合成数据效果不如真实样本子集。

Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?

论文配图:Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?
图 1 · 摘自论文原文
  • 用统一标准测试7种合成数据方法和3种精选数据策略
  • 合成数据在大模型上准确率低于甚至远差于精选数据
  • 精选数据更高效且能更好覆盖原始数据分布

数据蒸馏(DD)旨在通过将大规模数据集的信息压缩成少量合成样本,实现高效训练。然而,现有评估方法不一致,从标准ERM到单/多教师监督,难以区分蒸馏效果与评价方式的影响。许多研究声称蒸馏优于数据精简方法如核心集选择(CS),认为仅限于真实样本的子集会限制表达能力。本文通过大规模实验,在ImageNet-1K、ImageNet100和ImageNette上,对7种前沿DD方法和3种CS策略,在三种训练协议下进行基准测试。结果表明,部分DD方法甚至不及随机子集;顶尖DD方法在大规模数据集上表现与或劣于核心集,且构建成本显著更高。此外,评估显示核心集在代表性、多样性和质量上持续更优。这说明当前DD方法的实际优势有限,而核心集仍是更高效的数据中心学习方案。

原文摘要 · Abstract (English)

Dataset distillation (DD) has emerged as a prominent approach in data centric machine learning, aiming to synthesize compact training sets for efficient training by compressing the information in large datasets into a small number of synthetic samples. However, DD methods are often evaluated under inconsistent evaluation protocols, ranging from standard ERM to single/multi-teacher supervision, making it difficult to isolate the effectiveness of distilled data from evaluation. Moreover, many prior methods claim that DD outperforms data pruning approaches such as coreset selection (CS), based on the assumption that restricting condensed datasets to subsets of real samples fundamentally limits their expressiveness. In this work, we critically evaluate DD methods through large-scale experiments using standardized datasets and evaluation protocols to assess their intrinsic effectiveness. We benchmark seven state-of-the-art (SOTA) DD methods on ImageNet-1K, ImageNet100, and ImageNette, using three widely adopted training protocols against three CS strategies. Our results show that while some DD methods fail to outperform even simple random subsets, the SOTA DD approaches are comparable to or worse than coresets on large-scale datasets and incur a substantially higher cost for construction. Beyond accuracy, we also evaluate the representativeness, diversity, and quality of condensed sets, and find that coresets consistently achieve better coverage of the original data distribution. These findings highlight the limited practical advantages of current DD methods and show that coresets remain competitive and are often a more computationally efficient alternative for data-centric learning.

数据蒸馏核心集模型效率图像分类

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。