用尖峰时序统计解决神经网络权重传输难题,实现局部学习
Spike-based alignment learning solves the weight transport problem
- 通过尖峰时序统计动态修正突触不对称性
- 在概率神经网络中加速收敛至目标分布
- 适用于类脑硬件,兼容真实神经元波动
在机器学习与计算神经科学中,功能神经网络的可塑性常表现为代价函数上的梯度下降,但此类方法常带来对称性约束,难以满足生物网络或类脑硬件所需的局部计算。例如,基于玻尔兹曼分布的唤醒-睡眠学习要求连接对称,而误差反向传播则存在严重的权重传输问题。现有解决方案如反馈对齐虽规避了对称性需求,但随网络规模和深度增长而性能下降。本文提出尖峰基对齐学习(SAL),一种用于脉冲神经网络的互补学习规则,利用尖峰时序统计提取并校正有效互连间的不对称性。该机制完全基于尖峰信号且局部实现,并利用噪声增强鲁棒性。通过赫布与反赫布可塑性的协同作用,突触可恢复真实局部梯度,缓解由神经元与突触变异性带来的偏差——这在物理神经网络中普遍存在。实验表明:1)相比仅使用赫布可塑性,SAL显著提升概率脉冲网络收敛至目标分布的速度;2)在基于皮层微环路的神经层次结构中,SAL能有效对齐反馈权重,使反馈误差正确回传;3)本方法仅依赖局部可塑性即可在深层网络中实现有竞争力的性能。
原文摘要 · Abstract (English)
In both machine learning and in computational neuroscience, plasticity in functional neural networks is frequently expressed as gradient descent on a cost. Often, this imposes symmetry constraints that are difficult to reconcile with local computation, as is required for biological networks or neuromorphic hardware. For example, wake-sleep learning in networks characterized by Boltzmann distributions assumes symmetric connectivity. Similarly, the error backpropagation algorithm is notoriously plagued by the weight transport problem between the representation and the error stream. Existing solutions such as feedback alignment circumvent the problem by deferring to the robustness of these algorithms to weight asymmetry. However, they scale poorly with network size and depth. We introduce spike-based alignment learning (SAL), a complementary learning rule for spiking neural networks, which uses spike timing statistics to extract and correct the asymmetry between effective reciprocal connections. Apart from being spike-based and fully local, our proposed mechanism takes advantage of noise. Based on an interplay between Hebbian and anti-Hebbian plasticity, synapses can thereby recover the true local gradient. This also alleviates discrepancies that arise from neuron and synapse variability -- an omnipresent property of physical neuronal networks. We demonstrate the efficacy of our mechanism using different spiking network models. First, SAL can significantly improve convergence to the target distribution in probabilistic spiking networks versus Hebbian plasticity alone. Second, in neuronal hierarchies based on cortical microcircuits, SAL effectively aligns feedback weights to the forward pathway, thus allowing the backpropagation of correct feedback errors. Third, our approach enables competitive performance in deep networks using only local plasticity for weight transport.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。