用少量数据训练即可预测全量数据重要性，大幅降低模型训练成本
Effective Data Pruning through Score Extrapolation
- 基于小样本学习模式，用kNN和图神经网络外推全数据重要性分数
- 在4个数据集上验证，仅用10%数据训练即保持原有模型性能
- 适合需要高效数据筛选的场景，如大规模模型训练与数据清洗
训练先进机器学习模型需要海量数据，导致计算成本极高。现有数据剪枝方法通常需完成完整初始训练才能识别冗余样本，无法在单次训练中体现效率优势。为此，我们提出一种新颖的重要度分数外推框架，只需在极小数据子集上训练即可。该框架提出了两种初步方法：k近邻与图神经网络，通过从该最小子集中学习模式，准确预测整个数据集的样本重要性。我们在2种前沿剪枝方法（Dynamic Uncertainty 和 TDDS）、4个不同数据集（CIFAR-10、CIFAR-100、Places-365、ImageNet）以及3种训练范式（监督、无监督、对抗）下验证了该方法的有效性。结果表明，分数外推是扩展昂贵得分计算任务（如剪枝、数据归因等）的有前景方向。
原文摘要 · Abstract (English)
Training advanced machine learning models demands massive datasets, resulting in prohibitive computational costs. To address this challenge, data pruning techniques identify and remove redundant training samples while preserving model performance. Yet, existing pruning techniques predominantly require a full initial training pass to identify removable samples, negating any efficiency benefits for single training runs. To overcome this limitation, we introduce a novel importance score extrapolation framework that requires training on only a small subset of data. We present two initial approaches in this framework - k-nearest neighbors and graph neural networks - to accurately predict sample importance for the entire dataset using patterns learned from this minimal subset. We demonstrate the effectiveness of our approach for 2 state-of-the-art pruning methods (Dynamic Uncertainty and TDDS), 4 different datasets (CIFAR-10, CIFAR-100, Places-365, and ImageNet), and 3 training paradigms (supervised, unsupervised, and adversarial). Our results indicate that score extrapolation is a promising direction to scale expensive score calculation methods, such as pruning, data attribution, or other tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。