研究RBM中激活函数如何影响高阶关联学习能力
Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines
- 对比四种激活函数对权重统计特性的影响
- 发现仅指数型激活函数能有效学习强高阶关联分布
- 适用于研究神经网络表征能力与激活函数关系的学者
神经网络的成功主要源于大量参数与单个神经元输入输出间的非线性。本文研究受限玻尔兹曼机（RBMs）中权重统计特性与隐层单元非线性激活函数之间的关系，及其对二值可见节点分布的影响。针对线性、阶跃、ReLU和指数四种常见激活函数进行分析，定性预测模型学习可见节点间强高阶相互作用分布的能力。结果表明，在高斯权重的RBMs集合中，此类分布普遍稀少且难以学习，除非隐层激活函数为指数型。
原文摘要 · Abstract (English)
The great success of neural networks primarily arises from the presence of the large number of weight parameters combined with nonlinearities in the input-output relationship of single neurons. In this work, we study the relationship between the statistical properties of the weights and the nonlinearity of the hidden unit in Restricted Boltzmann Machines (RBMs) on the one side, and the distribution induced on binary visible units. We do this for four commonly used activation functions: Linear, Step, ReLU, and Exponential, and make qualitative predictions about the ability of these models to learn distributions with strong higher order interactions over the visible nodes. We show that in general, in an ensemble of RBMs with Gaussian weights, these distributions are rare and hard to learn, except when the hidden unit activation function is an Exponential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。