梯度下降通过神经元动态调整网络容量，让模型自动适应任务需求。
It's Not a Lottery, It's a Race: Understanding How Gradient Descent Adapts the Network's Capacity to the Task
- 分析单层ReLU网络中神经元的学习动态，发现三类关键机制。
- 训练后可通过合并相似神经元或剪掉低范数权重有效降低容量。
- 解释了彩票抽奖猜想的原理，揭示初始条件如何影响最终性能。
当前对神经网络的理论理解远落后于其实际表现。一个关键未解之谜是：为何且如何在梯度下降训练过程中，神经网络的理论容量会自发降低为与任务相匹配的有效容量。本文通过分析单隐层ReLU网络中个体神经元的学习动态，揭示了三种动力学原则——相互对齐、解封与竞争，共同解释了为何训练后常可成功压缩网络容量：例如合并等效神经元或剪除低范数权重。我们具体阐明了彩票抽奖猜想的机制，即某些神经元的特定有益初始条件为何能使其获得更高的权重范数。
原文摘要 · Abstract (English)
Our theoretical understanding of neural networks is lagging behind their empirical success. One of the important unexplained phenomena is why and how, during the process of training with gradient descent, the theoretical capacity of neural networks is reduced to an effective capacity that fits the task. We here investigate the mechanism by which gradient descent achieves this through analyzing the learning dynamics at the level of individual neurons in single hidden layer ReLU networks. We identify three dynamical principles, namely mutual alignment, unlocking and racing, that together explain why we can often successfully reduce capacity after training through the merging of equivalent neurons or the pruning of low norm weights. We specifically explain the mechanism behind the lottery ticket conjecture, or why the specific, beneficial initial conditions of some neurons lead them to obtain higher weight norms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。