将神经网络统一为连续形式，揭示其深层结构与近似误差关系。
A Unified Representation of Neural Networks Architectures
- 提出连续神经网络的积分表示，统一单层与深层残差网络
- 证明近似误差随神经元数量和层数增长而减小
- 适用于各类神经网络，尤其适合研究模型泛化与极限行为
本文研究当隐藏层神经元数量和隐藏层数量趋于无穷时神经网络的极限情况，形成连续体结构，并推导出近似误差与神经元数量及层数的关系。首先针对单隐藏层网络，构建广义的积分无限宽神经网络表示，推广了现有连续神经网络框架；进而扩展至具有有限积分隐藏层和残差连接的深层残差网络。其次，重新审视神经微分方程（Neural ODEs）与深层残差网络的关系，通过离散化技术量化逼近误差。最后，将两种方法融合，提出统一的同质化神经网络表示——分布式参数神经网络（DiPaNet），证明多数现有有限与无限维神经网络架构可通过均质化/离散化与该框架关联。本方法完全确定性，适用于任意一致连续的矩阵权函数。讨论了与神经场及其他神经积分微分方程的关系，并展望了该框架在更广泛场景中的应用潜力。
原文摘要 · Abstract (English)
In this paper we consider the limiting case of neural networks (NNs) architectures when the number of neurons in each hidden layer and the number of hidden layers tend to infinity thus forming a continuum, and we derive approximation errors as a function of the number of neurons and/or hidden layers. Firstly, we consider the case of neural networks with a single hidden layer and we derive an integral infinite width neural representation that generalizes existing continuous neural networks (CNNs) representations. Then we extend this to deep residual CNNs that have a finite number of integral hidden layers and residual connections. Secondly, we revisit the relation between neural ODEs and deep residual NNs and we formalize approximation errors via discretization techniques. Then, we merge these two approaches into a unified homogeneous representation of NNs as a Distributed Parameter neural Network (DiPaNet) and we show that most of the existing finite and infinite-dimensional NNs architectures are related via homogenization/discretization with the DiPaNet representation. Our approach is purely deterministic and applies to general, uniformly continuous matrix weight functions. Relations with neural fields and other neural integro-differential equations are discussed along with further possible generalizations and applications of the DiPaNet framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。