研究神经网络梯度流如何学习多指标函数，揭示收敛机制与时间复杂度。
Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence
- 用相关损失函数和梯度流拟合高维高斯数据上的多指标函数。
- 当指标向量正交时，神经元能以高概率收敛到真实方向，需约 k log k 个神经元。
- 当指标向量夹角过大时，相关损失失效，存在收敛失败的临界阈值。
本文研究在高维标准高斯数据上，使用相关损失函数的神经网络梯度流动态，以逼近多指标函数 $f^*(x) = \sum_{j=1}^k \sigma^*(v_j^T x)$，其中 $v_1, \dots, v_k$ 为单位向量，且 $\sigma^*$ 的Hermite展开中不含一阶和二阶多项式。已知单指标情形（$k=1$）需多项式时间克服搜索阶段。本文将该结果推广至任意方向的多指标函数。在搜索阶段后，神经元是否收敛至真实索引向量尚不明确。当索引向量正交时，我们完整刻画了固定点结构，并证明神经元会收敛到最近的索引向量。因此，当神经元数量 $n \asymp k \log k$ 时，梯度流几乎必然找到全部索引向量。当 $v_i^T v_j = \beta \geq 0$（$i \neq j$）时，我们证明存在一个锐利阈值 $\beta_c = c/(c+k)$，使得计算索引向量平均值的固定点从鞍点转变为最小值。数值模拟表明，采用相关损失和适度过参数化可有效学习近正交的索引向量，但当索引向量间内积超过某一阈值时，相关损失失效。
原文摘要 · Abstract (English)
This work focuses on the gradient flow dynamics of a neural network model that uses correlation loss to approximate a multi-index function on high-dimensional standard Gaussian data. Specifically, the multi-index function we consider is a sum of neurons $f^*(x) \!=\! \sum_{j=1}^k \! σ^*(v_j^T x)$ where $v_1, \dots, v_k$ are unit vectors, and $σ^*$ lacks the first and second Hermite polynomials in its Hermite expansion. It is known that, for the single-index case ($k\!=\!1$), overcoming the search phase requires polynomial time complexity. We first generalize this result to multi-index functions characterized by vectors in arbitrary directions. After the search phase, it is not clear whether the network neurons converge to the index vectors, or get stuck at a sub-optimal solution. When the index vectors are orthogonal, we give a complete characterization of the fixed points and prove that neurons converge to the nearest index vectors. Therefore, using $n \! \asymp \! k \log k$ neurons ensures finding the full set of index vectors with gradient flow with high probability over random initialization. When $ v_i^T v_j \!=\! β\! \geq \! 0$ for all $i \neq j$, we prove the existence of a sharp threshold $β_c \!=\! c/(c+k)$ at which the fixed point that computes the average of the index vectors transitions from a saddle point to a minimum. Numerical simulations show that using a correlation loss and a mild overparameterization suffices to learn all of the index vectors when they are nearly orthogonal, however, the correlation loss fails when the dot product between the index vectors exceeds a certain threshold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。