用生成模型初始化网络权重，提升小数据下两层神经网络的拟合能力。
Generative Feature Training of Thin 2-Layer Networks
- 用深度生成模型学习隐藏权重的先验分布并采样初始化
- 固定权重时输出层可解线性方程，加速训练收敛
- 在潜在空间微调并加正则化，有效避免噪声干扰
我们研究在小数据集和平方损失下，用少量隐藏权重的两层神经网络逼近函数的问题。由于能量景观高度非凸，梯度训练常陷入局部最优。为此，我们用一个参数化的深层生成模型，从学习到的提议分布中采样初始化隐藏权重。利用固定隐藏权重时最优输出权重可解线性方程的性质来训练该生成模型。生成模型训练完成后，我们在潜在空间中对采样权重进行梯度后处理，并引入正则化以抑制潜在噪声。最后通过数值实验验证了方法的有效性。
原文摘要 · Abstract (English)
We consider the approximation of functions by 2-layer neural networks with a small number of hidden weights based on the squared loss and small datasets. Due to the highly non-convex energy landscape, gradient-based training often suffers from local minima. As a remedy, we initialize the hidden weights with samples from a learned proposal distribution, which we parameterize as a deep generative model. To train this model, we exploit the fact that with fixed hidden weights, the optimal output weights solve a linear equation. After learning the generative model, we refine the sampled weights with a gradient-based post-processing in the latent space. Here, we also include a regularization scheme to counteract potential noise. Finally, we demonstrate the effectiveness of our approach by numerical examples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。