揭示了漏失ReLU两层网络梯度下降的定向收敛机制与良性过拟合条件。
Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networks
- 通过建立参数定向收敛，推导出分类误差上界。
- 在混合数据下证明良性过拟合在更广泛场景中以高概率成立。
- 首次在非近正交数据上实现理论突破，适合理论研究者参考。
本文为固定宽度漏失ReLU两层神经网络分类器在混合数据上通过梯度下降训练时的良性过拟合提供了充分条件。通过建立网络参数的定向收敛性及收敛方向的分类误差上界，我们发现了新的相变现象。此前，(漏失)ReLU网络中的定向收敛仅在梯度流中被证明；由于缺乏该性质，已有良性过拟合结果仅限于近正交数据。而我们的结论适用于更广泛的混合数据设置。实验表明，良性过拟合在远超以往已知范围的多种情形下以高概率发生。此外，我们还刻画了即使存在定向收敛，良性过拟合仍可能失效的情形。本工作为漏失ReLU两层网络中的良性过拟合提供了更完整的理论图景。
原文摘要 · Abstract (English)
In this paper, we provide sufficient conditions of benign overfitting of fixed width leaky ReLU two-layer neural network classifiers trained on mixture data via gradient descent. Our results are derived by establishing directional convergence of the network parameters and classification error bound of the convergent direction. Our classification error bound also lead to the discovery of a newly identified phase transition. Previously, directional convergence in (leaky) ReLU neural networks was established only for gradient flow. Due to the lack of directional convergence, previous results on benign overfitting were limited to those trained on nearly orthogonal data. All of our results hold on mixture data, which is a broader data setting than the nearly orthogonal data setting in prior work. We demonstrate our findings by showing that benign overfitting occurs with high probability in a much wider range of scenarios than previously known. Our results also allow us to characterize cases when benign overfitting provably fails even if directional convergence occurs. Our work thus provides a more complete picture of benign overfitting in leaky ReLU two-layer neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。