通过追踪神经元稀疏性，提出一种渐进式训练新方法。
Learning Neural Networks by Neuron Pursuit
- 从鞍点出发分析梯度流行为，发现小权重方向可稳定收敛。
- 实验验证：新算法在多个数据集上实现更高精度与更快收敛。
- 适合研究网络结构演化或追求高效训练的学者参考。
本文第一部分研究齐次神经网络在特定鞍点附近的梯度流演化，这些鞍点源于前人工作，是梯度流逃离原点后遇到的第一个鞍点。研究表明，当初始值足够接近此类鞍点时，梯度流将在较长一段时间内保持在鞍点附近，此时小范数权重集合始终保持较小，但其方向趋于收敛。第二部分基于此发现，提出一种名为神经元追踪（Neuron Pursuit, NP）的贪心训练算法，该算法通过迭代扩展网络（添加具有精心选择权重的神经元）并优化损失函数来训练深层网络。数值实验验证了该算法的有效性。
原文摘要 · Abstract (English)
The first part of this paper studies the evolution of gradient flow for homogeneous neural networks near a class of saddle points exhibiting a sparsity structure. The choice of these saddle points is motivated from previous works on homogeneous networks, which identified the first saddle point encountered by gradient flow after escaping the origin. It is shown here that, when initialized sufficiently close to such saddle points, gradient flow remains near the saddle point for a sufficiently long time, during which the set of weights with small norm remain small but converge in direction. Furthermore, important empirical observations are made on the behavior of gradient descent after escaping these saddle points. The second part of the paper, motivated by these results, introduces a greedy algorithm to train deep neural networks called Neuron Pursuit (NP). It is an iterative procedure which alternates between expanding the network by adding neuron(s) with carefully chosen weights, and minimizing the training loss using this augmented network. The efficacy of the proposed algorithm is validated using numerical experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。