分析高维经验风险局部极小值数量，揭示神经网络与统计模型的优化特性。
Local minima of the empirical risk in high dimension: General theorems and convex examples
- 基于高斯数据与投影损失结构，用Kac-Rice公式推导极小值期望数量。
- 在n∼d渐近下，证明极小值位置集中且误差有精确渐近表达式。
- 适用于凸与非凸损失，可分析最优解处海森矩阵谱结构，适合优化理论研究者。
我们研究一个高维经验风险最小化的一般模型：数据 $$\mathbf{x}_i$$ 是 $d$ 维高斯向量，模型参数为 $$\mathbfΘ\in\mathbb{R}^{d\times k}$$，损失函数通过投影 $$\mathbfΘ^\mathsf{T}\mathbf{x}_i$$ 依赖于数据。该设定涵盖经典统计方法（如多项回归等广义线性模型）以及含 $k$ 个隐藏单元的两层全连接神经网络。利用高斯过程理论中的 Kac-Rice 公式，在 $n,d\to\infty$ 且 $n\asymp d$ 的比例渐近下，推导出经验风险局部极小值期望数量的上界。通过马尔可夫不等式，可确定极小值位置（带指数偏差界），进而获得估计与预测误差的精确渐近表达式。作为特例，我们将结果应用于凸损失，证明该方法紧致并验证了先前的猜想。此外，还刻画了最优解处海森矩阵的谱结构。附录论文将此通用结果应用于非凸情形。
原文摘要 · Abstract (English)
We consider a general model for high-dimensional empirical risk minimization whereby the data $\mathbf{x}_i$ are $d$-dimensional Gaussian vectors, the model is parametrized by $\mathbfΘ\in\mathbb{R}^{d\times k}$, and the loss depends on the data via the projection $\mathbfΘ^\mathsf{T}\mathbf{x}_i$. This setting covers as special cases classical statistics methods (e.g. multinomial regression and other generalized linear models), but also two-layer fully connected neural networks with $k$ hidden neurons. We use the Kac-Rice formula from Gaussian process theory to derive a bound on the expected number of local minima of this empirical risk, under the proportional asymptotics in which $n,d\to\infty$, with $n\asymp d$. Via Markov's inequality, this bound allows to determine the positions of these minimizers (with exponential deviation bounds) and hence derive sharp asymptotics on the estimation and prediction error. As a special case, we apply our characterization to convex losses. We show that our approach is tight and allows to prove previously conjectured results. In addition, we characterize the spectrum of the Hessian at the minimizer. A companion paper applies our general result to non-convex examples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。