提出新型神经元可塑性机制，有效缓解神经网络学习新任务时遗忘旧知识的问题。
SNAP: Stopping Catastrophic Forgetting in Hebbian Learning with Sigmoidal Neuronal Adaptive Plasticity
- 采用分段式权重增长策略，模仿生物神经元的长时程增强机制
- 在赫布学习中完全避免旧任务遗忘，且仅需少量训练数据
- 适用于需要持续学习的场景，尤其适合类脑计算研究
人工神经网络（ANN）存在灾难性遗忘问题，即学习新任务时会严重遗忘旧任务。现有机器学习算法（包括基于随机梯度下降和赫布学习的方法）通常以线性方式随经验更新权重，与生物神经元行为不符。生物神经元在中等强度下具有高可塑性，但一旦达到一定强度便通过长时程增强（LTP）实现巩固。我们提出一种名为Sigmoidal Neuronal Adaptive Plasticity（SNAP）的人工近似方法，使权重呈现非线性增长特性，在达到足够大或小值时自动稳定。实验表明，在赫布学习中，SNAP可完全防止旧任务遗忘；而在基于SGD的学习中则无此效果。
原文摘要 · Abstract (English)
Artificial Neural Networks (ANNs) suffer from catastrophic forgetting, where the learning of new tasks causes the catastrophic forgetting of old tasks. Existing Machine Learning (ML) algorithms, including those using Stochastic Gradient Descent (SGD) and Hebbian Learning typically update their weights linearly with experience i.e., independently of their current strength. This contrasts with biological neurons, which at intermediate strengths are very plastic, but consolidate with Long-Term Potentiation (LTP) once they reach a certain strength. We hypothesize this mechanism might help mitigate catastrophic forgetting. We introduce Sigmoidal Neuronal Adaptive Plasticity (SNAP) an artificial approximation to Long-Term Potentiation for ANNs by having the weights follow a sigmoidal growth behaviour allowing the weights to consolidate and stabilize when they reach sufficiently large or small values. We then compare SNAP to linear weight growth and exponential weight growth and see that SNAP completely prevents the forgetting of previous tasks for Hebbian Learning but not for SGD-base learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。