arXiv:2502.06905cs.LGcs.AI2025-02ICML被引 26

提出DUAL评分,在训练初期快速筛选重要数据,大幅降低剪枝耗时。

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty

论文配图:Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
图 1 · 摘自论文原文
  • 基于样本难度和预测不确定性设计轻量级评分,无需完整训练即可剪枝。
  • 在ImageNet上90%剪枝率下仍达60%准确率,时间成本降至66%。
  • 适合追求高效数据清洗的实践者,尤其适用于噪声/损坏数据场景。

深度学习依赖大规模数据集,带来高昂存储与训练成本。数据剪枝旨在通过移除冗余样本缓解此问题,但现有方法常需完整训练多轮后才能剪枝,反而使剪枝过程比直接训练更昂贵。为此,本文提出一种难度与不确定性感知的轻量级(DUAL)评分机制,可在训练早期识别关键样本。为应对极端剪枝导致的性能骤降,进一步引入基于Beta分布的比例自适应采样策略。在多种数据集与学习场景(如含标签噪声、图像失真、模型架构泛化)下的实验表明,本方法优于此前最优方法。具体而言,在ImageNet-1k上,剪枝耗时仅为之前方法的66%,且在90%剪枝率下实现60%测试准确率;在CIFAR数据集上,耗时仅需15%的同时保持最优性能。

原文摘要 · Abstract (English)

Recent advances in deep learning rely heavily on massive datasets, leading to substantial storage and training costs. Dataset pruning aims to alleviate this demand by discarding redundant examples. However, many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset. To overcome this limitation, we introduce a Difficulty and Uncertainty-Aware Lightweight (DUAL) score, which aims to identify important samples from the early training stage by considering both example difficulty and prediction uncertainty. To address a catastrophic accuracy drop at an extreme pruning, we further propose a ratio-adaptive sampling using Beta distribution. Experiments on various datasets and learning scenarios such as image classification with label noise and image corruption, and model architecture generalization demonstrate the superiority of our method over previous state-of-the-art (SOTA) approaches. Specifically, on ImageNet-1k, our method reduces the time cost for pruning to 66% compared to previous methods while achieving a SOTA, specifically 60% test accuracy at a 90% pruning ratio. On CIFAR datasets, the time cost is reduced to just 15% while maintaining SOTA performance.

数据剪枝轻量化训练效率不确定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。