揭示深度模型在不平衡数据下的学习偏差,解释为何少数类难以泛化。
On the Impact of Class Imbalance on the Learning Dynamics of Deep Neural Networks:An Intuitive Insight
- 通过监控不同失衡比下模型对多数/少数类的学习模式,分析训练动态。
- 早期训练中模型严重欠拟合少数类,仅学习多数类,后期虽能学少数类但过拟合。
- 发现少数类表征非泛化,适合关注不平衡学习机制的研究者阅读。
深度神经网络(DNN)中的类别不平衡问题近年来受到广泛关注。然而,现有研究对不平衡导致性能下降的原因存在分歧,表明我们对这一长期现象如何影响DNN性能的理解仍不充分。本研究系统考察了类别不平衡对DNN学习动态的影响,通过监测模型在不同失衡比数据集上对多数类和少数类的学习模式。实验发现,与平衡数据上各分类学习相似的情况不同,类别不平衡会严重损害模型性能:在训练初期,模型对少数类样本严重欠拟合,同时仅学习多数类。尽管模型最终能学习少数类样本,但这种学习方式仅使少数类表示在测试阶段过拟合于降低整体训练损失,缺乏泛化能力。
原文摘要 · Abstract (English)
Class imbalance in deep neural networks (DNNs) has witnessed a rapid increase in research attention in recent years. However, the varying accounts of the reasons behind the poor performance of DNN on imbalance data in pertinent literature shows that little is known about how this agelong phenomenon impacts the performance of DNNs. A better understanding of this problem is crucial to developing effective DNN-based imbalance methods. Thus, this study systematically investigates the impact of class imbalance on the learning dynamics of DNN by monitoring the learning pattern of DNN models on both the majority and minority classes of datasets of varying imbalance ratios. Experimental findings shows that as against learning from balanced datasets where DNN learns the classes similarly, class imbalance has severe deteriorating impact on the performance of DNN, driving the model to underfit the minority class samples in the early training epochs while simultaneously learning only the majority class. Although DNN ultimately learns the minority samples, learning in this manner only results in learnt minority representations that are non-generalizable at test phase because they are merely overfitted to keep the overall training loss as low as possible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。