通过渐进式数据丢弃，训练速度提升87.6%且准确率更高
Progressive Data Dropout: An Embarrassingly Simple Approach to Faster Training
- 按难度分层丢弃数据，逐步减少训练轮次
- 仅需12.4%有效轮次，准确率反而提升4.82%
- 无需改架构或优化器，适配现有训练流程
机器学习的成功长期依赖大规模数据集训练，但模型与数据规模的双重增长带来巨大成本。当前训练仍采用均匀重复采样，效率低下。本文提出渐进式数据丢弃（Progressive Data Dropout），基于数据难易度分层筛选，显著减少有效训练轮次。实验表明，该方法将有效训练轮次降至基准的12.4%，且准确率最高提升4.82%。该方法不改变模型结构或优化器，可无缝集成至标准训练流程，具备广泛部署潜力。代码已开源：https://github.com/bazyagami/LearningWithRevision
原文摘要 · Abstract (English)
The success of the machine learning field has reliably depended on training on large datasets. While effective, this trend comes at an extraordinary cost. This is due to two deeply intertwined factors: the size of models and the size of datasets. While promising research efforts focus on reducing the size of models, the other half of the equation remains fairly mysterious. Indeed, it is surprising that the standard approach to training remains to iterate over and over, uniformly sampling the training dataset. In this paper we explore a series of alternative training paradigms that leverage insights from hard-data-mining and dropout, simple enough to implement and use that can become the new training standard. The proposed Progressive Data Dropout reduces the number of effective epochs to as little as 12.4% of the baseline. This savings actually do not come at any cost for accuracy. Surprisingly, the proposed method improves accuracy by up to 4.82%. Our approach requires no changes to model architecture or optimizer, and can be applied across standard training pipelines, thus posing an excellent opportunity for wide adoption. Code can be found here: https://github.com/bazyagami/LearningWithRevision
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。