揭示深度神经网络各层的逼近精度与深度的几何关系,为深层结构提供可解释的理论支撑。
Layer-wise Geometric Approximation Rates for Deep Networks
- 设计统一混合激活架构,使每层输出均可独立逼近目标函数
- 各层逼近误差控制在 $ (2d+1) imes N^{- heta} $ 的几何尺度内,$ θ $ 为层数
- 支持中途截断,适合需要渐进精度验证的应用场景
深度被认为是深度神经网络成功的关键因素,但传统近似理论仅对最终输出提供保证,对中间层的作用缺乏清晰解释。本文构建一个定量框架,使深度具有精确的尺度依赖性。我们设计了一个固定宽度 $2dN+d+2$、任意有限深度的共享混合激活网络,使得每一层读出 $Φ_ℓ$ 均可作为目标函数 $f$ 的逼近器。对于 $f ∈ L^p([0,1]^d)$ 且 $p ∈ [1,∞)$,第 $ℓ$ 层的逼近误差受 $L^p$ 模连续性在几何尺度 $N^{-ℓ}$ 处的值控制,系数为 $2d+1$。若 $f$ 为 1-利普希茨函数,误差率简化为 $(2d+1)N^{-ℓ}$。该网络设计受多级深度学习启发,深度作为逐步精化机制;对任意指定终端深度,构造出一组嵌套前缀读出,早期修正项保留在后续层中,可在达到所需认证精度后截断。
原文摘要 · Abstract (English)
Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear. We address this gap by developing a quantitative framework in which depth admits a precise scale-dependent interpretation. Specifically, we design a single shared mixed-activation architecture of fixed width $2dN+d+2$ and any prescribed finite depth such that each intermediate readout $Φ_\ell$ is itself an approximant to the target function $f$. For $f\in L^p([0,1]^d)$ with $p\in [1,\infty)$, the approximation error of $Φ_\ell$ is controlled by $(2d+1)$ times the $L^p$ modulus of continuity at the geometric scale $N^{-\ell}$ for all $\ell$. The estimate reduces to the geometric rate $(2d+1)N^{-\ell}$ if $f$ is $1$-Lipschitz. Our network design is inspired by multigrade deep learning, where depth serves as a progressive refinement mechanism. For every prescribed terminal depth, the construction yields a finite nested family of prefix readouts whose earlier correction terms remain embedded in later readouts. Thus the approximation may be truncated within the prescribed depth range once the desired certified accuracy is reached.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。