大参数量+大初始化下，神经网络损失曲面无伪极小值。
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
- 通过大宽度网络与大初始值，证明损失曲面无伪极小点。
- 需满足宽度m ≳ min(n^d, 2^n) 才能保证全局最优解存在。
- 适用于大初始化场景，小初始化仍可能陷入局部最优。
权重衰减在现代神经网络训练中广泛应用，但其理论理解仍不充分。本文研究两层ReLU网络在ℓ₂正则化下的损失景观。当网络宽度m满足m ≳ min(n^d, 2^n)，其中n为数据点数，d为输入维度时，损失景观趋于良性——即无伪极小值。在此条件下，几乎所有恒定激活区域均包含全局最小值且无非全局极小点。我们进一步证明该过参数化程度既充分也必要（以正交数据为例）。此外，此类结果主要存在于大初始化情形；而在小初始化（特征学习）下，优化仍可能收敛至伪极小值，即使整体景观全局良好。
原文摘要 · Abstract (English)
The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint. While weight decay is standard practice in modern training procedures, most theoretical analyses focus on unregularized settings. In this work, we investigate the loss landscape of the $\ell_2$-regularized training loss for two-layer ReLU networks. We show that the landscape becomes benign -- i.e., free of spurious local minima -- under large overparametrization, specifically when the network width $m$ satisfies $m \gtrsim \min(n^d, 2^n)$, where $n$ is the number of data points and $d$ the input dimension. More precisely in this regime, almost all constant activation regions contain a global minimum and no spurious local minima. We further show that this level of overparametrization is not only sufficient but also necessary via the example of orthogonal data. Finally, we demonstrate that such loss landscape results primarily hold relevance in the large initialization regime. In contrast, for small initializations -- corresponding to the feature learning regime -- optimization can still converge to spurious local minima, despite the global benignity of the landscape.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。