低相关数据下，小规模ReLU网络也能全局收敛。
Convergence of Shallow ReLU Networks on Weakly Interacting Data
- 利用高维输入的低相关性，证明宽为log(n)即可全局收敛。
- 收敛速率达1/n，数据正交时在1/n与1/√n间相变。
- 适合研究深度学习优化机理的学者参考。
我们分析了一维隐藏层ReLU网络在梯度流下对n个数据点的收敛性。核心贡献在于利用环境空间的高维性带来的输入样本低相关性，证明宽度约为log(n)的网络在高概率下即可实现全局收敛。分析采用Polyak-Łojasiewicz视角，沿梯度流轨迹得到收敛速率1/n。当数据完全正交时，进一步刻画收敛速度的渐近行为介于1/n与1/√n之间，并揭示收敛率存在相变现象，在相对时间尺度1/log(n)内从下界演化至上界。
原文摘要 · Abstract (English)
We analyse the convergence of one-hidden-layer ReLU networks trained by gradient flow on $n$ data points. Our main contribution leverages the high dimensionality of the ambient space, which implies low correlation of the input samples, to demonstrate that a network with width of order $\log(n)$ neurons suffices for global convergence with high probability. Our analysis uses a Polyak-Łojasiewicz viewpoint along the gradient-flow trajectory, which provides an exponential rate of convergence of $\frac{1}{n}$. When the data are exactly orthogonal, we give further refined characterizations of the convergence speed, proving its asymptotic behavior lies between the orders $\frac{1}{n}$ and $\frac{1}{\sqrt{n}}$, and exhibiting a phase-transition phenomenon in the convergence rate, during which it evolves from the lower bound to the upper, and in a relative time of order $\frac{1}{\log(n)}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。