解析两层神经网络在二分类中的双下降现象
The Double Descent Behavior in Two Layer Neural Network for Binary Classification
- 用凸高斯极小极大定理分析训练损失的全局解
- 模型复杂度增加时测试误差先降后升再降
- 适合研究泛化性能与模型规模关系的学者
近期研究发现了一个令人意外的现象：模型测试误差随复杂度变化呈现双下降趋势，即误差先降低后升高，随后再次下降。本文针对带有ReLU激活函数的两层神经网络，在监督学习框架下的二分类任务中，探究这一现象背后的数学机制。通过将模型规模定义为训练样本数与模型维度的比值，我们利用凸高斯极小极大定理，推导出全局训练损失的近似候选解，从而揭示双下降行为的理论基础。
原文摘要 · Abstract (English)
Recent studies observed a surprising concept on model test error called the double descent phenomenon, where the increasing model complexity decreases the test error first and then the error increases and decreases again. To observe this, we work on a two layer neural network model with a ReLU activation function designed for binary classification under supervised learning. Our aim is to observe and investigate the mathematical theory behind the double descent behavior of model test error for varying model sizes. We quantify the model size by the ratio of number of training samples to the dimension of the model. Due to the complexity of the empirical risk minimization procedure, we use the Convex Gaussian Min Max Theorem to find a suitable candidate for the global training loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。