基于记忆遗忘曲线,优化神经网络学习间隔以减少遗忘。
Do Your Best and Get Enough Rest for Continual Learning
- 按人类记忆规律设计学习-休息周期,调整重训间隔。
- 在多个任务上显著提升现有持续学习方法的性能。
- 适合需要长期稳定学习的场景,如在线模型更新。
根据艾宾浩斯遗忘曲线理论,通过大量学习并充分休息可增强记忆保留。为避免神经网络在持续学习中出现灾难性遗忘,本文提出视图批次(view-batch)模型,依据该理论调整学习调度,优化对相同样本重复训练的回忆间隔。该模型通过两种方法实现:1)一种保证最优回忆间隔的回放机制;2)一种从单个样本中逐步获取丰富知识的自监督学习策略。实验表明,该方法与遗忘曲线理论一致,能有效增强长期记忆。在多种协议和场景下,其显著提升了多个先进持续学习方法的表现。项目已开源:https://github.com/hankyul2/ViewBatchModel。
原文摘要 · Abstract (English)
According to the forgetting curve theory, we can enhance memory retention by learning extensive data and taking adequate rest. This means that in order to effectively retain new knowledge, it is essential to learn it thoroughly and ensure sufficient rest so that our brain can memorize without forgetting. The main takeaway from this theory is that learning extensive data at once necessitates sufficient rest before learning the same data again. This aspect of human long-term memory retention can be effectively utilized to address the continual learning of neural networks. Retaining new knowledge for a long period of time without catastrophic forgetting is the critical problem of continual learning. Therefore, based on Ebbinghaus' theory, we introduce the view-batch model that adjusts the learning schedules to optimize the recall interval between retraining the same samples. The proposed view-batch model allows the network to get enough rest to learn extensive knowledge from the same samples with a recall interval of sufficient length. To this end, we specifically present two approaches: 1) a replay method that guarantees the optimal recall interval, and 2) a self-supervised learning that acquires extensive knowledge from a single training sample at a time. We empirically show that these approaches of our method are aligned with the forgetting curve theory, which can enhance long-term memory. In our experiments, we also demonstrate that our method significantly improves many state-of-the-art continual learning methods in various protocols and scenarios. We open-source this project at https://github.com/hankyul2/ViewBatchModel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。