通过软权重缩放恢复神经网络学习新知识的能力
Recovering Plasticity of Neural Networks via Soft Weight Rescaling
- 每步训练后对权重进行温和缩放，防止权重无限增长
- 在图像分类任务上提升持续学习和热启动学习性能
- 无需重置网络，既保旧知识又恢复学习弹性
近期研究发现，随着训练进行，神经网络逐渐丧失学习新信息的能力，即塑性下降。其中，权重无界增长是主要诱因，还损害泛化能力并扰乱优化过程。重置网络虽可缓解此问题，但会丢失已学知识导致性能下降。本文提出软权重缩放（SWR），在不丢失信息的前提下防止权重无限增长。SWR通过在每一步学习中对权重进行缩放，恢复网络的可塑性。理论证明，SWR能控制权重大小并平衡层间权重幅度。实验表明，该方法在标准图像分类基准上，显著提升了热启动学习、持续学习及单任务学习的表现。
原文摘要 · Abstract (English)
Recent studies have shown that as training progresses, neural networks gradually lose their capacity to learn new information, a phenomenon known as plasticity loss. An unbounded weight growth is one of the main causes of plasticity loss. Furthermore, it harms generalization capability and disrupts optimization dynamics. Re-initializing the network can be a solution, but it results in the loss of learned information, leading to performance drops. In this paper, we propose Soft Weight Rescaling (SWR), a novel approach that prevents unbounded weight growth without losing information. SWR recovers the plasticity of the network by simply scaling down the weight at each step of the learning process. We theoretically prove that SWR bounds weight magnitude and balances weight magnitude between layers. Our experiment shows that SWR improves performance on warm-start learning, continual learning, and single-task learning setups on standard image classification benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。