arXiv:2411.08590cs.LG2024-11被引 8

统一了联想记忆模型,用新框架实现高效稀疏记忆检索。

Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval

  • 基于弗歇尔-杨损失构建能量函数,统一传统与现代霍普菲尔德网络。
  • 通过泰尔斯熵等方法实现端到端可微更新,支持稀疏变换和精确回忆。
  • 适用于图像、文本等多场景记忆任务,适合研究神经记忆与注意力机制者。

联想记忆模型(如霍普菲尔德网络及其现代变体)因记忆容量提升及与变换器中自注意力的关联而受到关注。本文提出统一框架——霍普菲尔德-弗歇尔-杨网络,将这些模型推广至更广泛的能量函数家族。能量函数定义为两个弗歇尔-杨损失之差:一个由广义熵参数化,用于霍普菲尔德评分机制;另一个对霍普菲尔德输出施加后处理变换。利用泰尔斯熵和范数熵,我们推导出端到端可微的更新规则,实现稀疏变换,并揭示损失边界、稀疏性与单模式精确回忆之间的新联系。进一步扩展该框架至结构化霍普菲尔德网络,结合稀疏最大后验(SparseMAP)变换,支持模式关联的检索而非单一模式。本框架统一并拓展了传统与现代霍普菲尔德网络,通过合适的弗歇尔-杨损失选择,以凸分析为基础,为ℓ₂归一化、层归一化等常见后处理提供能量最小化视角。我们在多种记忆回溯任务中验证了其有效性,包括自由回忆与序列回忆。实验在模拟数据、图像检索、多实例学习和文本解释任务上均证明了方法的有效性。

原文摘要 · Abstract (English)

Associative memory models, such as Hopfield networks and their modern variants, have garnered renewed interest due to advancements in memory capacity and connections with self-attention in transformers. In this work, we introduce a unified framework-Hopfield-Fenchel-Young networks-which generalizes these models to a broader family of energy functions. Our energies are formulated as the difference between two Fenchel-Young losses: one, parameterized by a generalized entropy, defines the Hopfield scoring mechanism, while the other applies a post-transformation to the Hopfield output. By utilizing Tsallis and norm entropies, we derive end-to-end differentiable update rules that enable sparse transformations, uncovering new connections between loss margins, sparsity, and exact retrieval of single memory patterns. We further extend this framework to structured Hopfield networks using the SparseMAP transformation, allowing the retrieval of pattern associations rather than a single pattern. Our framework unifies and extends traditional and modern Hopfield networks and provides an energy minimization perspective for widely used post-transformations like $\ell_2$-normalization and layer normalization-all through suitable choices of Fenchel-Young losses and by using convex analysis as a building block. Finally, we validate our Hopfield-Fenchel-Young networks on diverse memory recall tasks, including free and sequential recall. Experiments on simulated data, image retrieval, multiple instance learning, and text rationalization demonstrate the effectiveness of our approach.

联想记忆霍普菲尔德网络能量模型稀疏性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。