按实例学习状态动态停止训练,提升效率且不降精度。
Instance-dependent Early Stopping
- 基于损失值二阶差分判断实例是否学完,实现个体化停止。
- 实验显示可减少10%-50%反向传播计算量,加速训练过程。
- 适合追求训练效率的深度学习实践者,尤其适用于大模型训练。
在机器学习实践中,早停法常用于正则化模型并节省计算开销,通过在验证集性能不再提升时终止训练。然而传统早停对所有实例采用统一停止标准,忽略了个体学习状态差异,导致已充分学习的实例仍进行冗余计算。为此,本文提出实例依赖的早停(IES)方法,将早停机制从整个训练集细化到单个实例层面,核心思想是:一旦模型掌握某个实例,其训练即应停止。IES通过判断实例损失值的二阶差分是否稳定在零附近来判定其学习完成状态,相比直接使用损失值更一致可靠,从而可设定统一阈值决定是否停止反向传播。实验证明,排除已掌握实例后,梯度范数上升,训练损失下降更快,显著加速训练进程。在多个基准数据集上的实验表明,IES可减少10%-50%的反向传播实例数,同时保持或略微提升测试准确率与迁移学习性能。
原文摘要 · Abstract (English)
In machine learning practice, early stopping has been widely used to regularize models and can save computational costs by halting the training process when the model's performance on a validation set stops improving. However, conventional early stopping applies the same stopping criterion to all instances without considering their individual learning statuses, which leads to redundant computations on instances that are already well-learned. To further improve the efficiency, we propose an Instance-dependent Early Stopping (IES) method that adapts the early stopping mechanism from the entire training set to the instance level, based on the core principle that once the model has mastered an instance, the training on it should stop. IES considers an instance as mastered if the second-order differences of its loss value remain within a small range around zero. This offers a more consistent measure of an instance's learning status compared with directly using the loss value, and thus allows for a unified threshold to determine when an instance can be excluded from further backpropagation. We show that excluding mastered instances from backpropagation can increase the gradient norms, thereby accelerating the decrease of the training loss and speeding up the training process. Extensive experiments on benchmarks demonstrate that IES method can reduce backpropagation instances by 10%-50% while maintaining or even slightly improving the test accuracy and transfer learning performance of a model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。