通过动态生成权重实现任务间切换,缓解神经网络遗忘问题。
EWGN: Elastic Weight Generation and Context Switching in Deep Learning
- 用额外网络动态生成主网络权重,实现输入依赖的上下文切换。
- 在MNIST和Fashion-MNIST上验证,相比传统方法保留旧任务性能更优。
- 适合需要持续学习新任务且不丢弃旧知识的场景。
人类智能能学习并保留多种任务,这启发了通用人工智能的研究。持续学习方法正朝着这一目标迈进。然而,任务多样性与上下文切换对神经网络学习构成挑战,灾难性遗忘指在学习新任务时旧任务表现下降。通过在不同任务上下文间切换,可减少网络权重间的干扰。本文提出弹性权重生成网络(EWGN),用于在两个不同任务间进行上下文切换。该架构引入一个额外网络,动态生成主网络权重,并在学习过程中巩固已有权重。权重生成依赖输入,从而支持上下文切换。在标准计算机视觉数据集MNIST和Fashion-MNIST上,我们分析了全连接网络、卷积神经网络及EWGN架构在随机梯度下降和弹性权重固化算法下的旧任务表征保留能力。理解动态权重生成与上下文切换机制,有助于提升持续学习性能。
原文摘要 · Abstract (English)
The ability to learn and retain a wide variety of tasks is a hallmark of human intelligence that has inspired research in artificial general intelligence. Continual learning approaches provide a significant step towards achieving this goal. It has been known that task variability and context switching are challenging for learning in neural networks. Catastrophic forgetting refers to the poor performance on retention of a previously learned task when a new task is being learned. Switching between different task contexts can be a useful approach to mitigate the same by preventing the interference between the varying task weights of the network. This paper introduces Elastic Weight Generative Networks (EWGN) as an idea for context switching between two different tasks. The proposed EWGN architecture uses an additional network that generates the weights of the primary network dynamically while consolidating the weights learned. The weight generation is input-dependent and thus enables context switching. Using standard computer vision datasets, namely MNIST and fashion-MNIST, we analyse the retention of previously learned task representations in Fully Connected Networks, Convolutional Neural Networks, and EWGN architectures with Stochastic Gradient Descent and Elastic Weight Consolidation learning algorithms. Understanding dynamic weight generation and context-switching ability can be useful in enabling continual learning for improved performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。