用多值隐变量改进生成模型,提升结构化记忆能力。
The Gaussian-Multinoulli Restricted Boltzmann Machine: A Potts Model Extension of the GRBM
- 用q-state分类隐单元替代二元隐单元,扩展潜在空间表达力。
- 在相同容量下,回忆准确率优于或相当传统模型,训练成本相近。
- 适合需要离散结构推理的场景,如类比记忆与符号任务。
许多现实任务(如关联记忆、符号推理)需要离散且结构化的表示,而标准连续潜在模型难以表达。本文提出高斯-多项式受限玻尔兹曼机(GM-RBM),将高斯-伯努利RBM(GB-RBM)中的二元隐单元替换为q状态分类(泊茨)单元,从而获得更丰富的潜在状态空间以表达多值概念。我们给出了能量函数、条件分布及学习规则的完整推导,并详细说明了实用训练策略(对比发散配合温度退火和槽内多样性约束),以避免状态坍缩。为区分架构效应与单纯潜在容量的影响,我们在容量匹配和参数匹配两种设置下进行评估,将GM-RBM与配置为相同潜在分配数目的GB-RBM进行比较。在类比回忆和结构化记忆基准测试中,GM-RBM在同等容量下表现具有竞争力,甚至在多个情形下有所提升,且仅使用吉布斯更新,训练成本相近。该离散q元形式也便于高效实现。结果表明,对于可解析的RBMs,分类隐单元是二元隐单元的简单、可扩展替代方案。
原文摘要 · Abstract (English)
Many real-world tasks, from associative memory to symbolic reasoning, benefit from discrete, structured representations that standard continuous latent models can struggle to express. We introduce the Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM), a generative energy-based model that extends the Gaussian-Bernoulli RBM (GB-RBM) by replacing binary hidden units with q-state categorical (Potts) units, yielding a richer latent state space for multivalued concepts. We provide a self-contained derivation of the energy, conditional distributions, and learning rules, and detail practical training choices (contrastive divergence with temperature annealing and intra-slot diversity constraints) that avoid state collapse. To separate architectural effects from sheer latent capacity, we evaluate under both capacity-matched and parameter-matched setups, comparing GM-RBM with GB-RBM configured to have the same number of possible latent assignments. On analogical recall and structured memory benchmarks, GM-RBM achieves competitive, and in several regimes improved, recall at equal capacity with comparable training cost, despite using only Gibbs updates. The discrete q-ary formulation is also amenable to efficient implementation. These results clarify when categorical hidden units provide a simple, scalable alternative to binary latents for discrete inference within tractable RBMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。