快速跨数据集剪枝，提升自然语言任务微调效率
Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understanding
- 用TF-IDF与几何中位数快速评估样本重要性
- 小数据集保留远离中位数样本，大数据集分层距离剪枝
- 六组实验验证高效性，显著降低计算开销
数据集剪枝旨在选取数据子集以实现高效模型训练。尽管自然语言处理中的数据效率研究主要聚焦于预训练阶段的单语料库场景，但针对跨多样化数据集的任务特定微调，由于数据集规模、分布差异、类别不平衡及标签空间异质性，高效剪枝仍具挑战。现有跨数据集微调剪枝方法常依赖计算成本高昂的样本排序过程，通常需完整训练或参考模型。本文提出快速跨数据集剪枝（SCDP），通过TF-IDF嵌入结合几何中位数快速评估样本重要性，并采用适应数据集大小的剪枝策略：小数据集保留远离几何中位数的样本，大数据集则使用基于距离的分层剪枝。在六个不同数据集上的实验表明，该方法在多种任务与规模下均有效，显著减少计算资源消耗。代码已开源：https://github.com/he-y/NLP-Dataset-Pruning
原文摘要 · Abstract (English)
Dataset pruning aims to select a subset of a dataset for efficient model training. While data efficiency in natural language processing has primarily focused on within-corpus scenarios during model pre-training, efficient dataset pruning for task-specific fine-tuning across diverse datasets remains challenging due to variability in dataset sizes, data distributions, class imbalance and label spaces. Current cross-dataset pruning techniques for fine-tuning often rely on computationally expensive sample ranking processes, typically requiring full dataset training or reference models. We address this gap by proposing Swift Cross-Dataset Pruning (SCDP). Specifically, our approach uses TF-IDF embeddings with geometric median to rapidly evaluate sample importance. We then apply dataset size-adaptive pruning to ensure diversity: for smaller datasets, we retain samples far from the geometric median, while for larger ones, we employ distance-based stratified pruning. Experimental results on six diverse datasets demonstrate the effectiveness of our method, spanning various tasks and scales while significantly reducing computational resources. Source code is available at: https://github.com/he-y/NLP-Dataset-Pruning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。