提出可稳定训练的生物神经回路模型,突破传统RNN与脑模型的兼容难题。
Unconditional stability of a recurrent neural circuit implementing divisive normalization
- 用李雅普诺夫方法证明任意维电路在单位权矩阵下局部无条件稳定
- 发现电路等价于耦合阻尼谐振子,导出能量函数实现归一化目标
- 支持反向传播训练,性能优于其他神经动力学模型,适合生物启发研究
递归神经模型的稳定性是构建生物合理神经动力学模型的主要挑战。传统皮层回路模型因动态系统中的强非线性难以训练,优化问题受非线性稳定性约束困扰。而循环神经网络(RNN)虽擅长序列任务,却缺乏生物合理性与可解释性。本文将动态除法归一化(DN)与生物合理模型ORGaNICs关联,证明当递归权重矩阵为单位矩阵时,任意维的ORGaNICs电路具有无条件局部稳定性。该电路等价于耦合阻尼谐振子系统,由此推导出其能量函数,确立了电路及单个神经元的规范目标。对于一般递归权重矩阵,我们证明二维模型稳定,并实证验证高维稳定性。此外,由于内在稳定性与自适应时间常数,ORGaNICs可无需梯度裁剪或缩放进行通过时间反向传播训练,有效解决梯度爆炸、消失和振荡问题。在标准RNN基准测试中,ORGaNICs在静态图像分类任务上表现优于其他神经动力学模型,在序列任务上性能与LSTM相当。
原文摘要 · Abstract (English)
Stability in recurrent neural models poses a significant challenge, particularly in developing biologically plausible neurodynamical models that can be seamlessly trained. Traditional cortical circuit models are notoriously difficult to train due to expansive nonlinearities in the dynamical system, leading to an optimization problem with nonlinear stability constraints that are difficult to impose. Conversely, recurrent neural networks (RNNs) excel in tasks involving sequential data but lack biological plausibility and interpretability. In this work, we address these challenges by linking dynamic divisive normalization (DN) to the stability of ORGaNICs, a biologically plausible recurrent cortical circuit model that dynamically achieves DN and that has been shown to simulate a wide range of neurophysiological phenomena. By using the indirect method of Lyapunov, we prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit when the recurrent weight matrix is the identity. We thus connect ORGaNICs to a system of coupled damped harmonic oscillators, which enables us to derive the circuit's energy function, providing a normative principle of what the circuit, and individual neurons, aim to accomplish. Further, for a generic recurrent weight matrix, we prove the stability of the 2D model and demonstrate empirically that stability holds in higher dimensions. Finally, we show that ORGaNICs can be trained by backpropagation through time without gradient clipping/scaling, thanks to its intrinsic stability property and adaptive time constants, which address the problems of exploding, vanishing, and oscillating gradients. By evaluating the model's performance on RNN benchmarks, we find that ORGaNICs outperform alternative neurodynamical models on static image classification tasks and perform comparably to LSTMs on sequential tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。