用点式信息量筛选数据，少用数据也能高效训练出好模型。
Quality over Quantity: An Effective Large-Scale Data Reduction Strategy Based on Pointwise V-Information
- 基于点式信息量衡量样本难易，剔除简单样本提升数据质量。
- 删减10%-30%数据后，准确率仅下降0.0001%至0.76%。
- 适用于中英文NLP任务，加速训练且提升性能，适合数据优化场景。
为提升模型训练效率，数据驱动的人工智能亟需数据压缩。核心挑战在于从海量数据中选出最具指导性的样本，而非全量使用。本文提出基于点式信息量（PVI）的有效数据压缩策略。首先，利用PVI量化样本难度，剔除低难度样本，实验显示在删除10%-30%数据后，分类器准确率仅下降0.0001%至0.76%。其次，按递增PVI顺序进行渐进式训练，加速收敛，相比传统方法提升0.8%准确率。结果表明，结合高效数据压缩策略，训练最优子集可同时提升模型性能与训练效率。此外，我们已将原本仅限于英文数据的PVI框架扩展至多种中文自然语言处理任务与基础模型，实现了快速训练与跨语言数据压缩的有意义成果。
原文摘要 · Abstract (English)
In order to increase the effectiveness of model training, data reduction is essential to data-centric Artificial Intelligence (AI). It achieves this by locating the most instructive examples in massive datasets. To increase data quality and training efficiency, the main difficulty is choosing the best examples rather than the complete datasets. In this paper, we propose an effective data reduction strategy based on Pointwise V-Information (PVI). To enable a static method, we first use PVI to quantify instance difficulty and remove instances with low difficulty. Experiments show that classifier performance is maintained with only a 0.0001% to 0.76% decline in accuracy when 10%-30% of the data is removed. Second, we train the classifiers using a progressive learning strategy on examples sorted by increasing PVI, accelerating convergence and achieving a 0.8% accuracy gain over conventional training. Our findings imply that training a classifier on the chosen optimal subset may improve model performance and increase training efficiency when combined with an efficient data reduction strategy. Furthermore, we have adapted the PVI framework, which was previously limited to English datasets, to a variety of Chinese Natural Language Processing (NLP) tasks and base models, yielding insightful results for faster training and cross-lingual data reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。