解析高维损失曲面拓扑,揭示相变与优化机制
Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrieval
- 用Kac-Rice公式分析高维风险曲面的临界点分布
- 发现局部极小值处海森矩阵出现信号方向不稳定的相变
- 适用于相位恢复等高维统计模型,适合优化算法研究者
我们研究高维高斯单指标模型中经验风险最小化的损失曲面。目标是从依赖于标签对 $(\mathbf{x}_i \cdot \boldsymbolθ, \mathbf{x}_i \cdot \boldsymbolθ^\star)$ 的损失函数 $\hat{R}(\boldsymbolθ)$ 中恢复未知信号 $\boldsymbolθ^\star \in \mathbb{R}^d$($d \gg 1$),其中 $\mathbf{x}_i \sim \mathcal{N}(0, I_d)$,在样本量与维度成比例的渐近情形 $n \asymp d$ 下。基于Kac-Rice公式,我们分析了不同复杂度的临界点(包括局部极小值)的期望数量。结果表明,文献中已有的变分公式可被大幅简化,转化为仅含有限个标量参数的显式变分问题,可高效数值求解。该框架还预测了临界点的谱性质及标签联合分布。我们将其应用于真实相位恢复问题,推导出完整的拓扑相图,揭示了类似BBP的相变现象:局部极小值处的海森矩阵在信号方向上变得不稳定(由Kac-Rice公式预测)。通过与有限尺寸模拟对比,验证了分析对梯度流动力学的良好预测能力,准确捕捉了标签分布等细微特征。整体结果为高维统计模型的渐近曲面分析与拓扑平凡化现象开辟新路径。
原文摘要 · Abstract (English)
We consider the landscape of empirical risk minimization for high-dimensional Gaussian single-index models (generalized linear models). The objective is to recover an unknown signal $\boldsymbolθ^\star \in \mathbb{R}^d$ (where $d \gg 1$) from a loss function $\hat{R}(\boldsymbolθ)$ that depends on pairs of labels $(\mathbf{x}_i \cdot \boldsymbolθ, \mathbf{x}_i \cdot \boldsymbolθ^\star)_{i=1}^n$, with $\mathbf{x}_i \sim \mathcal{N}(0, I_d)$, in the proportional asymptotic regime $n \asymp d$. Using the Kac-Rice formula, we analyze different complexities of the landscape -- defined as the expected number of critical points -- corresponding to various types of critical points, including local minima. We first show that some variational formulas previously established in the literature for these complexities can be drastically simplified, reducing to explicit variational problems over a finite number of scalar parameters that we can efficiently solve numerically. Our framework also provides detailed predictions for properties of the critical points, including the spectral properties of the Hessian and the joint distribution of labels. We apply our analysis to the real phase retrieval problem for which we derive complete topological phase diagrams of the loss landscape, characterizing notably BBP-type transitions where the Hessian at local minima (as predicted by the Kac-Rice formula) becomes unstable in the direction of the signal. We test the predictive power of our analysis to characterize gradient flow dynamics, finding excellent agreement with finite-size simulations of local optimization algorithms, and capturing fine-grained details such as the empirical distribution of labels. Overall, our results open new avenues for the asymptotic study of loss landscapes and topological trivialization phenomena in high-dimensional statistical models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。