arXiv:2607.05546stat.MLcs.LG2026-07

提出统一函数空间理论,揭示深度与复杂度的深层关系。

Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

论文配图:Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity
图 1 · 摘自论文原文
  • 构建递归定义的函数空间,适用于多种激活函数。
  • 证明深度饱和:深层网络仅小幅缩放函数类,无新功能多样性。
  • 适合研究网络表达能力与复杂度的学者参考。

我们建立了全连接深度神经网络的统一函数空间理论。函数通过前一层激活函数的ℓ¹有界线性组合递归定义,第一层使用仿射函数字典。与仅适用于齐次激活(如ReLU)的现有理论不同,本框架为广泛使用的齐次与非齐次激活函数提供了有意义的功能复杂度度量。该简单构造统一了文献中看似无关的概念,包括基于范数的复杂度界和深度的变分表征,并促进了对范数约束下深层网络可表示函数的新分析。我们证明了该空间的新型表示定理,并建立了新的函数空间复杂度界,表明在任意深度下函数类仍保持定性小。在单变量ReLU情形下,我们证明了‘深度饱和’结果:深度仅带来常数级缩放,不增加功能多样性。因此,任意维度下受范数控制的深层ReLU函数无法沿任何方向表现出高频特征。这表明,一旦用函数空间范数而非参数量等代价控制复杂度,深度带来的常见表达优势便消失。总体而言,函数空间视角揭示了深度与复杂度之间的新结构关系。

原文摘要 · Abstract (English)

We develop a unified function space theory of deep fully connected neural networks. Functions in our spaces are defined recursively as $\ell^1$-bounded linear combinations of activated functions from preceding layers, with a dictionary of affine functions at the first layer. Unlike existing theories that are largely specialized to homogeneous activations such as the ReLU, our framework provides a meaningful notion of functional complexity for deep networks with a broad range of homogeneous and non-homogeneous activation functions commonly used in practice. This simple construction unites several seemingly disparate ideas from the literature, including norm-based complexity bounds and variational characterizations of depth, and facilitates novel analyses of what kinds of functions deep norm-constrained networks can represent. To this end, we prove a novel representer theorem for our spaces and establish novel function-space complexity bounds showing that the associated function classes remain qualitatively small at arbitrary depth. In the univariate ReLU case, we prove a "depth saturation" result: depth in this setting yields only a small constant rescaling of the function class, with no added functional diversity. As a consequence, we show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction. This finding reveals that some commonly cited expressivity benefits of depth disappear once network complexity is controlled by an appropriate function space norm, rather than parameter count or other representational costs that permit compounded rescaling across layers. Overall, our results illustrate how a function space perspective yields new structural insights into the relationship between depth and complexity.

深度学习函数空间表达能力复杂度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。