高维梯度下降的演化规律对数据分布不敏感,只要前两阶矩匹配即可。
Universality of high-dimensional scaling limits of stochastic gradient descent
- 用投影到低维子空间的损失函数建模高维学习任务,分析梯度下降动态
- 在数据为任意满足二阶矩匹配的乘积分布时,梯度下降收敛到相同的常微分方程
- 适用于理解神经网络训练动力学,尤其适合研究初始化与数据分布的影响
我们研究高维统计任务中,损失函数仅依赖于数据在由参数向量和真实向量张成的固定维子空间上的投影。这包括使用单层和双层网络进行混合分布分类(交叉熵损失)以及学习单指数和多指数模型。当数据来自各向同性高斯混合分布时,已知有限族摘要统计量在随机梯度下降下的演化会收敛到一个自治常微分方程(ODE),当维度和样本量趋于无穷且步长以合适速率趋于零时成立。本文主要结果是:这些 ODE 极限具有普遍性——只要数据来自任意乘积分布,其前两阶矩与高斯分布匹配,且初始化和真实向量坐标非局域化,则极限相同。我们还证明了两个非普遍性结果:若初始化坐标对齐,极限不再普遍;同时,围绕 ODE 平衡点的随机微分方程极限也不具普遍性。
原文摘要 · Abstract (English)
We consider statistical tasks in high dimensions whose loss depends on the data only through its projection into a fixed-dimensional subspace spanned by the parameter vectors and certain ground truth vectors. This includes classifying mixture distributions with cross-entropy loss with one and two-layer networks, and learning single and multi-index models with one and two-layer networks. When the data is drawn from an isotropic Gaussian mixture distribution, it is known that the evolution of a finite family of summary statistics under stochastic gradient descent converges to an autonomous ordinary differential equation (ODE), as the dimension and sample size go to $\infty$ and the step size goes to $0$ commensurately. Our main result is that these ODE limits are universal in that this limit is the same whenever the data is drawn from mixtures of arbitrary product distributions whose first two moments match the corresponding Gaussian distribution, provided the initialization and ground truth vectors are coordinate-delocalized. We complement this by proving two corresponding non-universality results. We provide a simple example where the ODE limits are non-universal if the initialization is coordinate aligned. We also show that the stochastic differential equation limits arising as fluctuations of the summary statistics around their ODE's fixed points are not universal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。