预训练能显著降低高维监督学习的样本需求，部分情况可实现指数级提升。
Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models
- 基于单指标模型分析预训练对梯度下降的影响
- 在高维场景下，样本复杂度降低多项式量级
- 某些情况下预训练比随机初始化提升指数级
无监督预训练和迁移学习常用于神经网络训练初始化，尤其在标注数据有限时。本文研究其对高维监督学习样本复杂度的影响。考虑通过在线随机梯度下降训练单层神经网络，在非常一般假设下证明：预训练与迁移学习（存在概念偏移）可使样本复杂度降低多项式因子（关于维度）。还揭示了一些令人惊讶的情况：预训练相比随机初始化，在样本复杂度上可实现指数级改进。
原文摘要 · Abstract (English)
Unsupervised pre-training and transfer learning are commonly used techniques to initialize training algorithms for neural networks, particularly in settings with limited labeled data. In this paper, we study the effects of unsupervised pre-training and transfer learning on the sample complexity of high-dimensional supervised learning. Specifically, we consider the problem of training a single-layer neural network via online stochastic gradient descent. We establish that pre-training and transfer learning (under concept shift) reduce sample complexity by polynomial factors (in the dimension) under very general assumptions. We also uncover some surprising settings where pre-training grants exponential improvement over random initialization in terms of sample complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。