重置权重比重置单元更能保持神经网络的持续学习能力。
Reinitializing weights vs units for maintaining plasticity in neural networks
- 提出选择性重置无用权重的新算法，提升网络可塑性。
- 小规模网络或含层归一化时，重置权重效果优于重置单元。
- 适用于需要长期学习的场景，如持续学习系统设计。
神经网络在持续学习非平稳数据时会出现可塑性下降问题，即丧失后续学习能力。本文对比了两种重置策略：重置网络单元与重置权重。提出一种新算法——选择性权重重置，专门重置网络中贡献最小的权重。在持续监督学习任务中实验发现，在网络单元数较少或包含层归一化时，重置权重比重置单元更有效维持可塑性；当网络足够大且不含层归一化时，两者效果相当。结果表明，权重重置在更多场景下具备更强适应性。
原文摘要 · Abstract (English)
Loss of plasticity is a phenomenon in which a neural network loses its ability to learn when trained for an extended time on non-stationary data. It is a crucial problem to overcome when designing systems that learn continually. An effective technique for preventing loss of plasticity is reinitializing parts of the network. In this paper, we compare two different reinitialization schemes: reinitializing units vs reinitializing weights. We propose a new algorithm, which we name \textit{selective weight reinitialization}, for reinitializing the least useful weights in a network. We compare our algorithm to continual backpropagation and ReDo, two previously proposed algorithms that reinitialize units in the network. Through our experiments in continual supervised learning problems, we identify two settings when reinitializing weights is more effective at maintaining plasticity than reinitializing units: (1) when the network has a small number of units and (2) when the network includes layer normalization. Conversely, reinitializing weights and units are equally effective at maintaining plasticity when the network is of sufficient size and does not include layer normalization. We found that reinitializing weights maintains plasticity in a wider variety of settings than reinitializing units.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。