将振荡神经元与阈值神经元耦合,实现新型记忆模型。
Binding threshold units with artificial oscillatory neurons
- 用李雅普诺夫函数约束动力学,统一建模两类神经元
- 振荡神经元可作为低秩修正项,提升记忆性能
- 该机制类比于神经网络的赫布学习或LoRA微调
人工柯尔莫哥洛夫振荡神经元被提出作为阈值单元的替代方案。实证表明,振荡单元在无监督目标发现和某些推理任务中表现更优。本文提出的耦合机制具有异质性,结合广义柯尔莫哥洛夫方程与传统阈值单元的耦合方法。本研究建立一个理论框架,明确区分振荡神经元与阈值神经元,并构建两者间的耦合机制。从生物角度,阈值单元表征神经元放电强度,而振荡单元通过频率调制实现信息交换。通过聚焦具有李雅普诺夫函数的动力系统,阈值单元导出霍普菲尔德关联记忆模型,振荡单元则导出特定形式的广义柯尔莫哥洛夫模型。由此产生的动力系统可自然耦合形成霍普菲尔德-柯尔莫哥洛夫关联记忆模型,同样具备李雅普诺夫函数。多种耦合形式可行,尤其可利用振荡神经元对霍普菲尔德网络权重矩阵进行低秩修正,该修正既可视为赫布学习,也可对应大语言模型微调中的流行方法LoRA。通过示例实验验证了该耦合的实际可行性。
原文摘要 · Abstract (English)
Artificial Kuramoto oscillatory neurons were recently introduced as an alternative to threshold units. Empirical evidence suggests that oscillatory units outperform threshold units in several tasks including unsupervised object discovery and certain reasoning problems. The proposed coupling mechanism for these oscillatory neurons is heterogeneous, combining a generalized Kuramoto equation with standard coupling methods used for threshold units. In this research note, we present a theoretical framework that clearly distinguishes oscillatory neurons from threshold units and establishes a coupling mechanism between them. We argue that, from a biological standpoint, oscillatory and threshold units realise distinct aspects of neural coding: roughly, threshold units model intensity of neuron firing, while oscillatory units facilitate information exchange by frequency modulation. To derive interaction between these two types of units, we constrain their dynamics by focusing on dynamical systems that admit Lyapunov functions. For threshold units, this leads to Hopfield associative memory model, and for oscillatory units it yields a specific form of generalized Kuramoto model. The resulting dynamical systems can be naturally coupled to form a Hopfield-Kuramoto associative memory model, which also admits a Lyapunov function. Various forms of coupling are possible. Notably, oscillatory neurons can be employed to implement a low-rank correction to the weight matrix of a Hopfield network. This correction can be viewed either as a form of Hebbian learning or as a popular LoRA method used for fine-tuning of large language models. We demonstrate the practical realization of this particular coupling through illustrative toy experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。