研究单变量两层ReLU网络在逻辑损失下的平坦性与泛化关系
Does Flatness imply Generalization for Logistic Loss in Univariate Two-Layer ReLU Network?
- 通过分析解的平坦性区域，给出泛化误差上界
- 发现存在无限平坦却过拟合的解，证明平坦性不保证泛化
- 理论结果在可控模拟中得到验证，适合关注泛化机制的研究者
我们研究单变量输入下任意过参数化的两层ReLU神经网络在逻辑损失下的泛化问题。近期研究表明，在平方损失下，平坦解（受平坦/稳定极小值和稳定边缘现象启发）可保证不发生过拟合，但逻辑损失下该现象是否成立仍不清楚。这构成一个令人困惑的开放问题，因为已有研究显示：梯度下降配合递增学习率会收敛到插值解（无穷远处，对可分边界情形）。本文证明：逻辑损失下，平坦性蕴含泛化更为微妙。正面结果：在每个候选解决定的左右‘不确定’集之间的区域内，平坦解具有近最优泛化界。负面结果：存在任意平坦却过拟合的解（位于无穷远），其处处被错误地判定为‘确定’，从而证明确保平坦性不足以保证泛化。我们的理论预测在精心控制的模拟实验中得以验证。
原文摘要 · Abstract (English)
We consider the problem of generalization of arbitrarily overparameterized two-layer ReLU Neural Networks with univariate input. Recent work showed that under square loss, flat solutions (motivated by flat / stable minima and Edge of Stability phenomenon) provably cannot overfit, but it remains unclear whether the same phenomenon holds for logistic loss. This is a puzzling open problem because existing work on logistic loss shows that gradient descent with increasing step size converges to interpolating solutions (at infinity, for the margin-separable cases). In this paper, we prove that the \emph{flatness implied generalization} is more delicate under logistic loss. On the positive side, we show that flat solutions enjoy near-optimal generalization bounds within a region between the left-most and right-most \emph{uncertain} sets determined by each candidate solution. On the negative side, we show that there exist arbitrarily flat yet overfitting solutions at infinity that are (falsely) certain everywhere, thus certifying that flatness alone is insufficient for generalization in general. We demonstrate the effects predicted by our theory in a well-controlled simulation study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。