研究深度增加时随机神经网络临界点数量的演化规律。
Critical Points of Random Neural Networks
- 基于激活函数的协方差导数，划分出三种临界点增长模式。
- 临界点数量可能收敛、多项式增长或指数增长，取决于激活函数性质。
- 为理解深层网络优化难度提供理论依据，适合优化与深度学习研究者。
本文研究了在无限宽极限下，不同激活函数的随机神经网络随深度增加时临界点的期望数量。在合适的正则性条件下，推导出固定索引临界点及超过给定阈值的临界点的精确渐近公式。分析揭示了三种不同行为：当协方差在1处的一阶导数取值不同时，临界点期望数量可能收敛、多项式增长或指数增长。理论预测得到数值实验验证。此外，我们还发现当正则性条件不满足（如使用ReLU激活函数）时，临界点数量随映射分辨率增大而上升，暗示临界点数量可能存在发散趋势。
原文摘要 · Abstract (English)
This work investigates the expected number of critical points of random neural networks with different activation functions as the depth increases in the infinite-width limit. Under suitable regularity conditions, we derive precise asymptotic formulas for the expected number of critical points of fixed index and those exceeding a given threshold. Our analysis reveals three distinct regimes depending on the value of the first derivative of the covariance evaluated at 1: the expected number of critical points may converge, grow polynomially, or grow exponentially with depth. The theoretical predictions are supported by numerical experiments. Moreover, we provide numerical evidence suggesting that, when the regularity condition is not satisfied (e.g. for neural networks with ReLU as activation function), the number of critical points increases as the map resolution increases, indicating a potential divergence in the number of critical points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。