研究神经网络遗忘与结构坍塌的关系，发现二者强相关。
Why Do Neural Networks Forget: A Study of Collapse in Continual Learning
- 通过权重和激活有效秩衡量模型结构坍塌程度。
- 不同方法在任务准确率和结构保持上表现差异显著。
- 适合关注模型内部机制的持续学习研究者阅读。
灾难性遗忘是持续学习中的主要问题，现有方法多基于任务准确率评估，忽视了模型内部结构。近期研究表明，结构坍塌导致可塑性下降，表现为有效秩（eRank）变化。这暗示了遗忘机制：网络无法扩展特征空间以学习新任务，被迫覆盖已有表征。为此，本研究通过测量权重和激活的有效秩，探究遗忘与结构坍塌的相关性。在Split MNIST和Split CIFAR-100基准上，评估了MLP、ConvGRU、ResNet-18和Bi-ConvGRU四种架构，分别采用SGD、LwF和经验回放（ER）策略训练。结果表明，遗忘与结构坍塌强相关，不同持续学习策略在保持模型容量与性能方面效率各异。
原文摘要 · Abstract (English)
Catastrophic forgetting is a major problem in continual learning, and lots of approaches arise to reduce it. However, most of them are evaluated through task accuracy, which ignores the internal model structure. Recent research suggests that structural collapse leads to loss of plasticity, as evidenced by changes in effective rank (eRank). This indicates a link to forgetting, since the networks lose the ability to expand their feature space to learn new tasks, which forces the network to overwrite existing representations. Therefore, in this study, we investigate the correlation between forgetting and collapse through the measurement of both weight and activation eRank. To be more specific, we evaluated four architectures, including MLP, ConvGRU, ResNet-18, and Bi-ConvGRU, in the split MNIST and Split CIFAR-100 benchmarks. Those models are trained through the SGD, Learning-without-Forgetting (LwF), and Experience Replay (ER) strategies separately. The results demonstrate that forgetting and collapse are strongly related, and different continual learning strategies help models preserve both capacity and performance in different efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。