重采样权重可加速模型适应新任务，优化器选择影响学习与遗忘的动态过程。
How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks
- 通过重采样最后一层权重，提升模型在持续学习中的适应速度。
- 实验表明，重采样后模型在新任务迁移时恢复更快，遗忘更少。
- 优化器的选择会引发任务间协同或干扰的复杂模式，影响多任务学习效果。
近期研究发现，在神经网络最后层重采样权重（即「zapping」）具有有益效果。尽管实证结果已证明其有效性，但其内在机制仍不明确。本文深入研究了卷积神经网络在持续学习和少量样本迁移学习等挑战性设置下，学习与遗忘的内部模式，使用手写字符与自然图像数据集进行实验。结果显示，训练中经过zapping的模型在迁移到新领域时能更快恢复。为进一步观察多任务持续学习的影响，我们量化了每个任务的受扰程度。结果表明，不仅zapping本身，优化器的选择也深刻影响学习与遗忘的动态，导致任务间出现复杂的协同或干扰模式。
原文摘要 · Abstract (English)
Recent work in continual learning has highlighted the beneficial effect of resampling weights in the last layer of a neural network (``zapping"). Although empirical results demonstrate the effectiveness of this approach, the underlying mechanisms that drive these improvements remain unclear. In this work, we investigate in detail the pattern of learning and forgetting that take place inside a convolutional neural network when trained in challenging settings such as continual learning and few-shot transfer learning, with handwritten characters and natural images. Our experiments show that models that have undergone zapping during training more quickly recover from the shock of transferring to a new domain. Furthermore, to better observe the effect of continual learning in a multi-task setting we measure how each individual task is affected. This shows that, not only zapping, but the choice of optimizer can also deeply affect the dynamics of learning and forgetting, causing complex patterns of synergy/interference between tasks to emerge when the model learns sequentially at transfer time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。