用可学习通道注意力的浅层网络，以更少数据高效学习球面低次多项式。
Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attention
- 设计带通道注意力的两层网络，通过可学习机制自动识别目标函数的真实阶数。
- 训练所需样本量仅为 $Θ(d^{\ell_0}/\eps)$，显著优于传统方法 $Θ(d^{\ell_0} \max\{\eps^{-2},\log d\})$。
- 首次实现有限宽度网络在特征学习下达到极小最大风险最优率，适合高维低次函数建模研究。
本文研究在 $\RR^d$ 单位球面上定义的低阶球面多项式（阶数 $\ell_0 = Θ(1) \ge 1$）的泛化学习问题。针对过参数化的两层神经网络，引入通道注意力机制，提出一种分阶段训练策略：第一阶段通过可学习的通道选择算法识别真实阶数 $\ell_0$；第二阶段使用标准梯度下降训练第二层。结果表明，在任意回归风险 $\eps \in (0,1)$ 下，该网络以高概率仅需 $n \asymp Θ(d^{\ell_0}/\eps)$ 的样本量即可完成学习，远低于经典复杂度 $Θ\pth{d^{\ell_0} \max\set{\eps^{-2},\log d}}$。同时，其非参数回归风险达到 $Θ(d^{\ell_0}/n)$ 的紧致上界，与秩为 $Θ(d^{\ell_0})$ 的核方法的极小最大风险率一致，即达到极小最大最优。这是首次在有限宽度、具备特征学习能力的过参数化网络中实现该最优率。
原文摘要 · Abstract (English)
We study the problem of learning a low-degree spherical polynomial of degree $\ell_0 = Θ(1) \ge 1$ defined on the unit sphere in $\RR^d$ by training an over-parameterized two-layer neural network (NN) with channel attention in this paper. Our main result is the significantly improved sample complexity for learning such low-degree polynomials. We show that, for any regression risk $\eps \in (0,1)$, a carefully designed two-layer NN with channel attention and finite width trained by the vanilla gradient descent (GD) requires the lowest sample complexity of $n \asymp Θ(d^{\ell_0}/\eps)$ with high probability, in contrast with the representative sample complexity $Θ\pth{d^{\ell_0} \max\set{\eps^{-2},\log d}}$, where $n$ is the training data size. Moreover, such sample complexity is not improvable since the trained network renders a sharp rate of the nonparametric regression risk of the order $Θ(d^{\ell_0}/{n})$ with high probability. On the other hand, the minimax optimal rate for the regression risk with a kernel of rank $Θ(d^{\ell_0})$ is $Θ(d^{\ell_0}/{n})$, so that the rate of the nonparametric regression risk of the network trained by GD is minimax optimal. Training the two-layer NN with channel attention proceeds in two stages: (1) a provable learnable channel selection algorithm, as a learnable harmonic-degree selection process, identifies the ground truth channel number in the target function, $\ell_0$, from $L \ge \ell_0$ channels in the first-layer activation; (2) the second layer is trained by standard GD using the selected channels. To the best of our knowledge, this is the first time a minimax optimal risk bound is obtained by training an over-parameterized but finite-width neural network with feature learning capability to learn low-degree spherical polynomials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。