arXiv:2409.07708stat.MLcond-mat.dis-nn2024-09被引 3

为受限玻尔兹曼机设计无需数据的权重初始化方法,提升学习效率。

Dataset-Free Weight-Initialization on Restricted Boltzmann Machine

  • 基于统计力学分析,从零均值高斯分布中采样权重,标准差由层间相关性最大化决定。
  • 在玩具和真实数据集上验证,新方法显著提升受限玻尔兹曼机的学习效率。
  • 适用于无训练数据时初始化玻尔兹曼机,尤其适合小样本或数据受限场景。

在前馈神经网络中,已有如LeCun、Xavier(或Glorot)和He等无需数据的权重初始化方法,它们基于特定分布(如高斯或均匀分布)随机设定权重参数,无需使用训练数据。据作者所知,目前尚无针对受限玻尔兹曼机(RBMs)——一种由两层构成的概率神经网络——的此类方法。本文基于统计力学分析,推导出适用于伯努利-伯努利型受限玻尔兹曼机的无数据权重初始化方法。该方法将权重从零均值高斯分布中抽取,其标准差通过假设‘能提升两层间层相关性(LC)的标准差可提高学习效率’进行优化,且层相关性的表达式由统计力学推导得出,最优标准差对应于层相关性最大点。在特定情况下(两层大小相同、层变量为{-1,1}二值、所有偏置为零),该方法与Xavier初始化一致。通过玩具数据集和真实数据集的数值实验验证了该方法的有效性。

原文摘要 · Abstract (English)

In feed-forward neural networks, dataset-free weight-initialization methods such as LeCun, Xavier (or Glorot), and He initializations have been developed. These methods randomly determine the initial values of weight parameters based on specific distributions (e.g., Gaussian or uniform distributions) without using training datasets. To the best of the authors' knowledge, such a dataset-free weight-initialization method is yet to be developed for restricted Boltzmann machines (RBMs), which are probabilistic neural networks consisting of two layers. In this study, we derive a dataset-free weight-initialization method for Bernoulli--Bernoulli RBMs based on statistical mechanical analysis. In the proposed weight-initialization method, the weight parameters are drawn from a Gaussian distribution with zero mean. The standard deviation of the Gaussian distribution is optimized based on our hypothesis that a standard deviation providing a larger layer correlation (LC) between the two layers improves the learning efficiency. The expression of the LC is derived based on a statistical mechanical analysis. The optimal value of the standard deviation corresponds to the maximum point of the LC. The proposed weight-initialization method is identical to Xavier initialization in a specific case (i.e., when the sizes of the two layers are the same, the random variables of the layers are $\{-1,1\}$-binary, and all bias parameters are zero). The validity of the proposed weight-initialization method is demonstrated in numerical experiments using a toy and real-world datasets.

权重初始化受限玻尔兹曼机统计力学无数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。