用连续时间记忆压缩霍普菲尔德网络，提升存储效率。
Modern Hopfield Networks with Continuous-Time Memories
- 将离散记忆转为连续时间密度表示，更新规则更高效。
- 在合成与视频数据上保持性能，计算成本显著降低。
- 符合人类工作记忆的资源分配理论，适合神经机制研究者。
近期研究建立了现代霍普菲尔德网络（HNs）与Transformer注意力头之间的联系，并保证了指数级存储容量。然而，这些模型在高效扩展存储方面仍面临挑战。受工作记忆中连续神经资源分配的心理学理论启发，我们提出一种将大型离散霍普菲尔德记忆压缩为更小、连续时间记忆的方法。通过引入连续注意力，新能量函数修改了霍普菲尔德网络的更新规则，用连续记忆上的概率密度替代传统的基于softmax的概率质量函数。这一公式与现代人类执行功能观点一致，为工作记忆中的吸引子动力学与资源高效记忆分配之间提供了原则性联系。该框架在保持与霍普菲尔德网络相当性能的同时，实现了压缩记忆，降低了合成数据集和视频数据集上的计算开销。
原文摘要 · Abstract (English)
Recent research has established a connection between modern Hopfield networks (HNs) and transformer attention heads, with guarantees of exponential storage capacity. However, these models still face challenges scaling storage efficiently. Inspired by psychological theories of continuous neural resource allocation in working memory, we propose an approach that compresses large discrete Hopfield memories into smaller, continuous-time memories. Leveraging continuous attention, our new energy function modifies the update rule of HNs, replacing the traditional softmax-based probability mass function with a probability density, over the continuous memory. This formulation aligns with modern perspectives on human executive function, offering a principled link between attractor dynamics in working memory and resource-efficient memory allocation. Our framework maintains competitive performance with HNs while leveraging a compressed memory, reducing computational costs across synthetic and video datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。