深度神经网络可高效逼近各向异性和混合光滑函数,突破维度灾难限制。
Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks
- 基于ReLU网络的宽度与深度,提出各向异性与混合光滑函数的新逼近率。
- 对各向异性函数逼近率达$(WL)^{-2\tilde{s}}$,对混合光滑函数达$(WL)^{-2s}$(含对数因子)。
- 适用于高维光滑函数建模,尤其适合需突破维度瓶颈的研究场景。
本文研究深度ReLU神经网络在$ L^p([0,1]^d) $范数下对光滑函数的逼近与学习效率。已有工作在Sobolev嵌入条件$s/d>1/q-1/p$下证明了对于Besov空间$\mathcal{B}^s_{q,r}([0,1]^d)$,逼近率为$\mathcal{O}((WL)^{-2s/d})$。为克服该速率中的维度灾难,本文将结果拓展至各向异性与混合光滑函数类。对于各向异性Besov空间$\mathcal{B}^{\boldsymbol{s}}_{q,r}([0,1]^d)$,在均值光滑度$\tilde{s} = (\sum_{i=1}^d s_i^{-1})^{-1}$满足$\tilde{s} > 1/q-1/p$时,建立逼近率$\mathcal{O}((WL)^{-2\tilde{s}})$;对于混合光滑Besov空间$\mathcal{MB}^s_{q,r}([0,1]^d)$,当$s>1/q-1/p$时,逼近率可达$\mathcal{O}((WL)^{-2s})$(含对数因子)。进一步推导了各向异性函数复合的逼近界,并证明深度ReLU网络在广泛光滑函数类上可达到最小最大最优率(对数因子内)。
原文摘要 · Abstract (English)
This paper studies how efficiently deep ReLU neural networks can approximate and learn smooth functions. When the error is measured in $L^p([0,1]^d)$ norm and the approximator is a network with width $W$ and depth $L$, recent works have proven the supper approximation rate $\mathcal{O}((WL)^{-2s/d})$ for Besov space $\mathcal{B}^s_{q,r}([0,1]^d)$ under the Sobolev embedding condition $s/d>1/q-1/p$. In order to overcome the curse of dimensionality in this rate, we extent this result to anisotropic and mixed smooth function classes. We establish the approximation rate $\mathcal{O}((WL)^{-2\tilde{s}})$ for anisotropic Besov space $\mathcal{B}^{\boldsymbol{s}}_{q,r}([0,1]^d)$ with anisotropic smoothness $\boldsymbol{s}=(s_1,\dots,s_d)$ under the embedding condition $\tilde{s} > 1/q-1/p$, where the mean smoothness $\tilde{s} = (\sum_{i=1}^d s_i^{-1})^{-1}$. For mixed smooth Besov space $\mathcal{MB}^s_{q,r}([0,1]^d)$ with mixed smoothness $s>1/q-1/p$, we show that the approximation rate $\mathcal{O}((WL)^{-2s})$ holds up to logarithmic factors. Using these results, we also derive approximation bounds for the composition of anisotropic Besov functions. As an application, it is shown that deep ReLU neural networks can achieve minimax optimal rates up to logarithmic factors for a wide range of smooth function classes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。