用p进数理论揭示深度网络的临界组织机制,连接层级结构与相变现象。
Critical Organization of Deep Neural Networks, and p-Adic Statistical Field Theories
- 用p进整数构建网络层级拓扑,将深度网络转化为p进树结构
- 在参数空间临界点处,网络状态从唯一态分裂为无穷多态
- 提出可计算输出概率分布的随机模型,揭示幂律展开特征
本文严格研究了深度神经网络(DNNs)和循环神经网络(RNNs)在激活函数为Sigmoid时的热力学极限。热力学极限对应一个连续神经网络,其中神经元构成无限点的连续空间。我们证明,在参数空间的某一区域内,该网络具有唯一且对参数连续依赖的状态;而在该区域外,状态分裂为无穷多个。这种状态突变称为临界组织,即网络从单一稳定态向无穷多态的分岔。我们利用p进整数编码层次结构,并提出算法将DNN和RNN的层级拓扑重构为p进树状结构。在此框架下,层级组织与临界组织相互关联。我们对一个基于p进细胞神经网络的灰度图像边缘检测玩具模型进行了严格分析,发现其临界组织可描述为奇异吸引子。第二部分研究随机版本的DNN和RNN,其中网络参数为二次可积函数空间中的广义高斯随机变量。在无限宽度极限下,我们计算了给定输入时输出的概率分布,发现其具有幂律展开形式,常数项为高斯分布。
原文摘要 · Abstract (English)
We rigorously study the thermodynamic limit of deep neural networks (DNNS) and recurrent neural networks (RNNs), assuming that the activation functions are sigmoids. A thermodynamic limit is a continuous neural network, where the neurons form a continuous space with infinitely many points. We show that such a network admits a unique state in a certain region of the parameter space, which depends continuously on the parameters. This state breaks into an infinite number of states outside the mentioned region of parameter space. Then, the critical organization is a bifurcation in the parameter space, where a network transitions from a unique state to infinitely many states. We use p-adic integers to codify hierarchical structures. Indeed, we present an algorithm that recasts the hierarchical topologies used in DNNs and RNNs as p-adic tree-like structures. In this framework, the hierarchical and the critical organizations are connected. We study rigorously the critical organization of a toy model, a hierarchical edge detector for grayscale images based on p-adic cellular neural networks. The critical organization of such a network can be described as a strange attractor. In the second part, we study random versions of DNNs and RNNs. In this case, the network parameters are generalized Gaussian random variables in a space of quadratic integrable functions. We compute the probability distribution of the output given the input, in the infinite-width case. We show that it admits a power-type expansion, where the constant term is a Gaussian distribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。