神经网络在极少量样本下高效学习高维特征,逼近理论极限。
Neural Networks Learn Generic Multi-Index Models Near Information-Theoretic Limit
- 通过分层梯度下降,两层网络可自动提取隐藏子空间。
- 仅需约 d 个样本、d² 时间即达信息论最优性能。
- 适合研究表示学习与高效深度学习的科研人员参考。
在深度学习中,理解神经网络如何高效学习高维特征是核心问题。本文研究标准两层神经网络在梯度下降下的学习行为,针对一般高斯多指标模型 $f(oldsymbol{x})=g(oldsymbol{U}oldsymbol{x})$,其中隐藏子空间 $oldsymbol{U} otin ext{R}^{r imes d}$。在链接函数满足非退化条件下,证明该网络可通过分层梯度下降实现 $o_d(1)$ 的测试误差,仅需 $ ilde{ ext{O}}(d)$ 样本和 $ ilde{ ext{O}}(d^2)$ 时间。样本与时间复杂度均与信息论极限一致,因此达到最优。训练初期,内层权重执行类似幂迭代的过程,隐式完成对隐藏子空间的谱初始化,消除有限样本噪声并恢复其结构。结果表明,只有当第一层训练超过 $ ext{O}(1)$ 步时,才能实现最优性能。该工作展示了神经网络在样本与时间效率上学习层级函数的能力。
原文摘要 · Abstract (English)
In deep learning, a central issue is to understand how neural networks efficiently learn high-dimensional features. To this end, we explore the gradient descent learning of a general Gaussian Multi-index model $f(\boldsymbol{x})=g(\boldsymbol{U}\boldsymbol{x})$ with hidden subspace $\boldsymbol{U}\in \mathbb{R}^{r\times d}$, which is the canonical setup to study representation learning. We prove that under generic non-degenerate assumptions on the link function, a standard two-layer neural network trained via layer-wise gradient descent can agnostically learn the target with $o_d(1)$ test error using $\widetilde{\mathcal{O}}(d)$ samples and $\widetilde{\mathcal{O}}(d^2)$ time. The sample and time complexity both align with the information-theoretic limit up to leading order and are therefore optimal. During the first stage of gradient descent learning, the proof proceeds via showing that the inner weights can perform a power-iteration process. This process implicitly mimics a spectral start for the whole span of the hidden subspace and eventually eliminates finite-sample noise and recovers this span. It surprisingly indicates that optimal results can only be achieved if the first layer is trained for more than $\mathcal{O}(1)$ steps. This work demonstrates the ability of neural networks to effectively learn hierarchical functions with respect to both sample and time efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。