深度神经网络的几何结构随层数变化呈现分形或规则特征。
Fractal and Regular Geometry of Deep Neural Networks
- 通过分析激活函数在不同深度下的边界体积，揭示网络几何演化规律。
- 非规则激活函数下边界体积呈分形，豪斯多夫维数随深度单调上升。
- 规则激活函数下边界体积行为由单一谱参数决定，可预测收敛、恒定或指数发散。
我们通过研究随机神经网络在不同激活函数下随深度增加的游走集边界体积，分析其几何特性。对于不规则激活函数（如Heaviside阶跃函数），边界体积表现出分形行为，其豪斯多夫维数随深度单调递增。而对于更规则的激活函数（如ReLU、logistic和tanh），随着深度增加，期望边界体积可能收敛至零、保持恒定或指数发散，具体取决于一个可轻松计算的单一谱参数。理论结果在基于蒙特卡洛模拟的数值实验中得到验证。
原文摘要 · Abstract (English)
We study the geometric properties of random neural networks by investigating the boundary volumes of their excursion sets for different activation functions, as the depth increases. More specifically, we show that, for activations which are not very regular (e.g., the Heaviside step function), the boundary volumes exhibit fractal behavior, with their Hausdorff dimension monotonically increasing with the depth. On the other hand, for activations which are more regular (e.g., ReLU, logistic and $\tanh$), as the depth increases, the expected boundary volumes can either converge to zero, remain constant or diverge exponentially, depending on a single spectral parameter which can be easily computed. Our theoretical results are confirmed in some numerical experiments based on Monte Carlo simulations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。