高阶关联网络在稀疏模式下实现超多项式存储容量
On associative neural networks for sparse patterns with huge capacities
- 将高阶相互作用引入稀疏联想记忆模型
- 固定阶数时容量为系统规模的多项式增长，阶数对数增长则达超多项式容量
- 适用于大规模稀疏数据存储，适合神经网络架构设计者
具有高阶或指数相互作用项的广义霍普菲尔德模型相比经典二次模型有显著更高的存储容量。而针对稀疏模式的联想记忆模型（如Willshaw和Amari模型）在稀疏情形下已优于经典霍普菲尔德模型。本文结合这两种机制，提出稀疏联想记忆模型的高阶版本，并研究其存储容量。对于固定相互作用阶数n，容量呈系统规模的多项式增长；当阶数随神经元数量对数增长时，容量达到超多项式级别。我们还讨论了在Gripon–Berrou架构中的类似情况，该架构原用于非稀疏信息（见文献[1]）。结果表明，高阶相互作用带来的容量提升在稀疏设置中依然存在，但具体存储尺度依赖于底层架构。
原文摘要 · Abstract (English)
Generalized Hopfield models with higher-order or exponential interaction terms are known to have substantially larger storage capacities than the classical quadratic model. On the other hand, associative memories for sparse patterns, such as the Willshaw and Amari models, already outperform the classical Hopfield model in the sparse regime. In this paper we combine these two mechanisms. We introduce higher-order versions of sparse associative memory models and study their storage capacities. For fixed interaction order $n$, we obtain storage capacities of polynomial order in the system size. When the interaction order is allowed to grow logarithmically with the number of neurons, this yields super-polynomial capacities. We also discuss an analogue in the Gripon--Berrou architecture which was formulated for non-sparse messages (see \cite{griponc}). Our results show that the capacity increase caused by higher-order interactions persists in the sparse setting, although the precise storage scale depends on the underlying architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。