分析ReLU网络中第二层神经元稳定不激活的概率规律
Stably unactivated neurons in ReLU neural networks
- 从对称分布初始化权重，推导第二层神经元不激活概率公式
- 当第一层有n₀+1个神经元时，概率为(2^{n₀}+1)/4^{n₀+1}
- 适用于理解网络表达能力受限的机制，适合研究神经网络设计者
神经网络架构的选择影响其可实现函数的范围，因此研究架构的表达能力备受关注。在ReLU神经网络中，稳定不激活的神经元会降低网络表达能力。本文研究当权重和偏置从对称概率分布初始化时，第二隐藏层神经元稳定不激活的概率。对于输入维度为n₀的网络，若第一隐藏层有n₀+1个神经元，则该概率精确为(2^{n₀}+1)/4^{n₀+1}；若第一隐藏层有n₁个神经元且n₁ ≤ n₀，概率为1/2^{n₁+1}。当第一隐藏层神经元多于n₀+1时，提出一个猜想并给出计算证据支持。
原文摘要 · Abstract (English)
The choice of architecture of a neural network influences which functions will be realizable by that neural network and, as a result, studying the expressiveness of a chosen architecture has received much attention. In ReLU neural networks, the presence of stably unactivated neurons can reduce the network's expressiveness. In this work, we investigate the probability of a neuron in the second hidden layer of such neural networks being stably unactivated when the weights and biases are initialized from symmetric probability distributions. For networks with input dimension $n_0$, we prove that if the first hidden layer has $n_0+1$ neurons then this probability is exactly $\frac{2^{n_0}+1}{4^{n_0+1}}$, and if the first hidden layer has $n_1$ neurons, $n_1 \le n_0$, then the probability is $\frac{1}{2^{n_1+1}}$. Finally, for the case when the first hidden layer has more neurons than $n_0+1$, a conjecture is proposed along with the rationale. Computational evidence is presented to support the conjecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。