证明了残差网络在内层宽为1时，最小块宽的理论下界。
Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One
- 研究残差网络在内层宽为1时的逼近能力，给出块宽上下界。
- 在 $L^p$ 精度下，最小块宽为 $\max\{d_x,d_y\}$，统一逼近需 $\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}$。
- 无论内层宽度如何，块宽小于 $\max\{d_x,d_y\}$ 则无法逼近某些函数，适合理论研究者。
本文研究残差神经网络的通用逼近性质。对于输入维度 $d_x$、输出维度 $d_y$，以及 LeakyReLU、ReLU、ReLU-like 激活函数，建立了最小块宽的上下界。在任意紧集上实现 $L^p$ 逼近（$1\leq p < +\infty$）时，若每个残差分支内层宽为1，则最小块宽精确为 $\max\{d_x,d_y\}$。此外，在相同约束下，块宽为 $\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}$ 的残差网络可实现统一逼近。进一步证明，对任意激活函数族，均存在函数无法被块宽小于 $\max\{d_x,d_y\}$ 的残差网络逼近，无论内层宽度如何，该结论在 $L^p$ 和一致逼近意义下均成立。因此，当激活函数为 LeakyReLU、ReLU、ReLU-like 且 $d_y\geq 2d_x+1$ 时，统一逼近的最小块宽为 $d_y$。
原文摘要 · Abstract (English)
In this paper, we study the universal approximation property of residual neural networks. For input and output dimensions $d_x$ and $d_y$, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the minimum block width are established. To achieve $L^p$ approximation $(1\leq p <+\infty)$ on any compact set, we show that the exact minimum block width is $\max\{d_x,d_y\}$ when each residual branch has inner width 1. Furthermore, we show that residual neural networks with block width $\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}$ can achieve uniform approximation on any compact set under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that there exist functions that cannot be approximated by residual neural networks with block width less than $\max\{d_x, d_y\}$, both in the $L^p$ sense and the uniform sense, regardless of inner width. Consequently, for LeakyReLU, ReLU, ReLU-like activation functions and $d_y\geq 2d_x+1$, the exact minimum block width for uniform approximation is $d_y$ when each residual branch has inner width 1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。