用更少参数实现高维函数的任意精度逼近，且参数量与维度线性相关。
Optimal Neural Network Approximation for High-Dimensional Continuous Functions
- 基于改进的柯尔莫哥洛夫叠加定理，构建由两个小网络重复组成的复合结构。
- 仅需约10889d个非零参数即可实现超逼近，参数量随维度线性增长。
- 证明了参数量下限为d，表明该方法在理论上已最优，适合高维建模研究者。
近期研究者提出一种宽度为 $36d(2d + 1)$、深度为 11 的神经网络，使用特殊激活函数——基本通用激活函数，可在任意精度下逼近 $[a,b]^d$ 上的连续函数，且仅需 $ ilde{ ext{O}}(d^2)$ 个神经元。本文通过变体柯尔莫哥洛夫叠加定理，构造出由基本通用激活函数生成的复合网络，其非零参数最多为 $10889d + 10887$，由两个5层网络重复组成：一个宽度为 $36(2d+1)$，另一个宽度为36。此外，我们证明存在一类连续函数，至少需要 $d$ 宽度（即至少 $d$ 个参数）才能实现任意精度逼近，表明该方法中参数量在维度 $d$ 下已达到理论最优，优于部分指数级增长的方法。
原文摘要 · Abstract (English)
Recently, the authors of \cite{SYZ22} developed a neural network with width $36d(2d + 1)$ and depth $11$, which utilizes a special activation function called the elementary universal activation function, to achieve the super approximation property for functions in $C([a,b]^d)$. That is, the constructed network only requires a fixed number of neurons (and thus parameters) to approximate a $d$-variate continuous function on a $d$-dimensional hypercube with arbitrary accuracy. More specifically, only $\mathcal{O}(d^2)$ neurons or parameters are used. One natural question is whether we can reduce the number of these neurons or parameters in such a network. By leveraging a variant of the Kolmogorov Superposition Theorem, \textcolor{black}{we show that there is a composition of networks generated by the elementary universal activation function with at most $10889d + 10887$ nonzero parameters such that this super approximation property is attained. The composed network consists of repeated evaluations of two neural networks: one with width $36(2d+1)$ and the other with width 36, both having 5 layers.} Furthermore, we present a family of continuous functions that requires at least width $d$, and thus at least $d$ neurons or parameters, to achieve arbitrary accuracy in its approximation. This suggests that the number of nonzero parameters is optimal in the sense that it grows linearly with the input dimension $d$, unlike some approximation methods where parameters may grow exponentially with $d$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。