揭示非ReLU激活函数如何影响神经网络核的性质。
Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks
- 分析了以零点不光滑为特征的激活函数对核空间的影响。
- 发现多数常见激活函数生成的核空间在不同深度下等价。
- 适用于研究初始化时无限宽网络的平滑性，适合理论研究者。
近年来，神经正切核（NTK）和神经网络高斯过程核（NNGP）为全连接神经网络提供了可分析的极限情形。然而，对于除幂形式ReLU外的激活函数，这些核的性质仍不清晰。本文的核心贡献是刻画了激活函数仅在零点处不光滑时，其对应的再生核希尔伯特空间（RKHS）的结构。该结果扩展了现有理论至SELU、ELU、LeakyReLU等多种常用激活函数。此外，我们还分析了缺失偏置、两层网络、多项式激活等特殊情形。结果表明，一类非无限光滑的激活函数在不同网络深度下生成等价的RKHS，其等价性仅取决于非光滑度的阶数；而多项式激活函数生成的RKHS则依赖于网络深度。最后，我们推导出关于NNGP样本路径平滑性的结果，刻画了初始化时无限宽神经网络的光滑性。
原文摘要 · Abstract (English)
In recent years, the neural tangent kernel (NTK) and neural network Gaussian process kernel (NNGP) have given theoreticians tractable limiting cases of fully connected neural networks. However, the property of these kernels are poorly understood for activation functions other than powers of the ReLU. Our main contribution is a characterization of the RKHS of these kernels for activation functions whose only non-smoothness is at zero. This extends existing theory to numerous commonly used activation functions such as SELU, ELU, or LeakyReLU. Additionally, we analyze a broad set of special cases such as missing biases, two-layer networks, or polynomial activations. Our results show that a broad class of not infinitely smooth activations generate equivalent RKHSs at different network depths, depending only on the degree of the non-smoothness up to equivalence. On the other hand, the RKHS generated by polynomial activations depends on the network depth. Finally, we derive results for the smoothness of NNGP sample paths, characterizing the smoothness of infinitely wide neural networks at initialization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。