用数学方法揭示神经网络如何记与忘,统一解释记忆形成与灾难性遗忘。
Memorisation and forgetting in a learning Hopfield neural network: bifurcation mechanisms, attractors and basins
- 通过分析81神经元霍普菲尔德网络的分岔机制,揭示记忆形成的动态过程。
- 学习过程中出现鞍结分岔,导致新记忆产生和旧记忆突然消失。
- 该方法适用于各类循环神经网络,对理解模型缺陷有重要启发。
尽管基于人工神经网络(ANN)的人工智能飞速发展,但其运作机制仍被视为“黑箱”,难以理解学习过程中如何形成记忆或产生不良特征,如虚假记忆和灾难性遗忘。尽管已有大量研究关注ANN学习的孤立方面,但因其高维度和非线性特性,全面分析仍具挑战。在ANN中,知识被认为存在于连接权重或吸引子盆地中,但这两者未被明确关联。本文对一个81神经元的霍普菲尔德网络在赫布学习下的记忆形成机制进行了系统分析,揭示了导致吸引子形成与破坏的分岔过程及其盆地边界演化。结果显示,施加刺激通过改变连接权重,引发叉式分岔及一系列鞍结分岔,生成新的吸引子及其对应盆地,可编码真实或虚假记忆;同时导致旧记忆突然消失(灾难性遗忘)。成功学习后,新类别由新生点吸引子的盆地表示,其边界由新鞍点的稳定流形构成。因此,记忆与遗忘是同一机制的两种表现。该分析策略具有普适性,适用于任意形式的循环神经网络。所揭示的记忆形成与灾难性遗忘机制,为更广泛类别的循环神经网络提供了洞见,并有助于开发缓解其缺陷的方法。
原文摘要 · Abstract (English)
Despite explosive expansion of artificial intelligence based on artificial neural networks (ANNs), these are employed as "black boxes'', as it is unclear how, during learning, they form memories or develop unwanted features, including spurious memories and catastrophic forgetting. Much research is available on isolated aspects of learning ANNs, but due to their high dimensionality and non-linearity, their comprehensive analysis remains a challenge. In ANNs, knowledge is thought to reside in connection weights or in attractor basins, but these two paradigms are not linked explicitly. Here we comprehensively analyse mechanisms of memory formation in an 81-neuron Hopfield network undergoing Hebbian learning by revealing bifurcations leading to formation and destruction of attractors and their basin boundaries. We show that, by affecting evolution of connection weights, the applied stimuli induce a pitchfork and then a cascade of saddle-node bifurcations creating new attractors with their basins that can code true or spurious memories, and an abrupt disappearance of old memories (catastrophic forgetting). With successful learning, new categories are represented by the basins of newly born point attractors, and their boundaries by the stable manifolds of new saddles. With this, memorisation and forgetting represent two manifestations of the same mechanism. Our strategy to analyse high-dimensional learning ANNs is universal and applicable to recurrent ANNs of any form. The demonstrated mechanisms of memory formation and of catastrophic forgetting shed light on the operation of a wider class of recurrent ANNs and could aid the development of approaches to mitigate their flaws.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。