用范数约束的神经网络高效学习稀疏组合函数,避开高维诅咒。
Learning Sparse Compositional Functions with Norm-Constrained Neural Networks

- 用参数弗罗贝尼乌斯范数控制复杂度,分析过参数情形下的逼近能力。
- 在狄利克雷有向无环图结构下,实现与维度无关的误差收敛速率。
- 适用于可有效图灵计算的函数,对多指标模型等架构通用性强。
深度神经网络具备学习分层特征的能力,被认为是其在高维学习中成功的关键机制。现有理论部分支持这一观点,通过参数数量和样本复杂度建立了无维数诅咒(CoD)的组合模型近似率。为研究参数量超过样本数的过参数化情形,本文提出以参数范数衡量复杂度的新框架。基于该框架,我们针对由有向无环图(DAGs)表示的稀疏组合函数,使用弗罗贝尼乌斯范数约束的深层神经网络,推导出逼近率与超出风险界。结果表明,深度网络可有效利用目标函数的组合结构,通过分层表示规避维数诅咒。由于所有能高效图灵计算的函数均可表示为稀疏组合形式,本结果具有广泛适用性,涵盖多指标模型、二叉树结构及一般组合架构。
原文摘要 · Abstract (English)
The ability of deep neural networks to learn hierarchical features is widely regarded as a key mechanism underlying their success in high-dimensional learning. Existing theory partially supports this view by establishing approximation rates based on parameter counts and sample complexity guarantees for compositional models without incurring the curse of dimensionality (CoD). To study overparameterized regimes, where the number of parameters exceeds the sample size, we develop a framework that measures complexity via the parameter norm. Within this approach, we establish approximation rates and excess risk bounds for learning sparse compositional functions whose compositional structure is represented by directed acyclic graphs (DAGs), using Frobenius norm-constrained deep neural networks. Our results have broad applicability since every function that is efficiently Turing computable admits sparse compositional representations. In particular, we cover a range of representative models, including multi-index models, binary tree structures, and general compositional architectures. The rates we derive show that deep networks can exploit the compositional structure of the target functions, effectively avoiding the CoD through hierarchical representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。