零均值激活函数让大宽神经网络输出近乎独立，或成可解释性研究新基准。
Wide Neural Networks as a Baseline for the Computational No-Coincidence Conjecture
- 用零均值非线性激活函数的大宽网络，输出几乎相互独立。
- ReLU、GeLU需加偏移才满足条件，纯形式不满足零均值要求。
- 为对齐研究中心的不可巧合猜想提供可计算的神经网络基准。
我们证明：在自然超参数设置下，随机初始化的大宽度神经网络，其输出近乎独立，当且仅当激活函数在标准正态分布下期望为零。例如，带偏移的ReLU和GeLU，以及tanh均满足此条件，而纯ReLU或GeLU则不满足。由于输出近似独立的特性，我们建议将零均值激活函数的神经网络作为对齐研究中心提出的‘计算不可巧合猜想’的候选基准——该猜想旨在衡量AI可解释性的极限。
原文摘要 · Abstract (English)
We establish that randomly initialized neural networks, with large width and a natural choice of hyperparameters, have nearly independent outputs exactly when their activation function is nonlinear with zero mean under the Gaussian measure: $\mathbb{E}_{z \sim \mathcal{N}(0,1)}[σ(z)]=0$. For example, this includes ReLU and GeLU with an additive shift, as well as tanh, but not ReLU or GeLU by themselves. Because of their nearly independent outputs, we propose neural networks with zero-mean activation functions as a promising candidate for the Alignment Research Center's computational no-coincidence conjecture -- a conjecture that aims to measure the limits of AI interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。