研究了神经网络逃离原点后的梯度流动态，发现权重方向和稀疏结构会保持不变。
Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin
- 分析局部Lipschitz梯度下网络逃离原点后的训练轨迹
- 首次揭示了逃逸后首个鞍点的特性
- 证明权重稀疏结构在逃逸前后保持稳定
近期研究指出，在小初始化条件下，同质神经网络在训练初期权重接近原点且方向趋于收敛。本文进一步研究了具有局部Lipschitz梯度的同质神经网络在逃离原点后的梯度流动态。基于此分析，刻画了梯度流在逃逸原点后遇到的第一个鞍点的特性。此外，对于同质前馈神经网络，在特定条件下，权重在逃逸前形成的稀疏结构可在逃逸后持续保持，直至抵达下一个鞍点。
原文摘要 · Abstract (English)
Recent works exploring the training dynamics of homogeneous neural network weights under gradient flow with small initialization have established that in the early stages of training, the weights remain small and near the origin, but converge in direction. Building on this, the current paper studies the gradient flow dynamics of homogeneous neural networks with locally Lipschitz gradients, after they escape the origin. Insights gained from this analysis are used to characterize the first saddle point encountered by gradient flow after escaping the origin. Also, it is shown that for homogeneous feed-forward neural networks, under certain conditions, the sparsity structure emerging among the weights before the escape is preserved after escaping the origin and until reaching the next saddle point.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。