先删无用数据再压缩，让小样本更高效
Distill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-Value-Based Pruning
- 先按损失值筛选重要样本，再进行数据蒸馏
- 删掉80%数据后仍提升5.2%准确率
- 适合需要跨模型泛化的高效训练场景
数据蒸馏近年来受到广泛关注，但现有方法通常基于完整数据集，可能包含无效样本。本文提出「先剪枝、后蒸馏」新框架，通过损失值采样系统性地修剪数据集，再结合经典蒸馏技术和生成先验，构建出具有代表性的核心子集。该方法显著提升蒸馏质量，在删除80%原始数据的前提下，准确率最高提升5.2个百分点，同时增强对未见架构的泛化能力，解决了当前蒸馏方法在跨模型适用性上的难题。实验结果表明，该方法在优先保留易样本的同时具备强鲁棒性，为高质量、高效率的数据蒸馏提供了新路径。
原文摘要 · Abstract (English)
Dataset distillation has gained significant interest in recent years, yet existing approaches typically distill from the entire dataset, potentially including non-beneficial samples. We introduce a novel "Prune First, Distill After" framework that systematically prunes datasets via loss-based sampling prior to distillation. By leveraging pruning before classical distillation techniques and generative priors, we create a representative core-set that leads to enhanced generalization for unseen architectures - a significant challenge of current distillation methods. More specifically, our proposed framework significantly boosts distilled quality, achieving up to a 5.2 percentage points accuracy increase even with substantial dataset pruning, i.e., removing 80% of the original dataset prior to distillation. Overall, our experimental results highlight the advantages of our easy-sample prioritization and cross-architecture robustness, paving the way for more effective and high-quality dataset distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。