初始权重影响神经网络专一性，进而决定遗忘程度。
A Theory of Initialisation's Impact on Specialisation
- 通过理论分析发现初始权重不平衡和高熵利于特征专一化
- 非专一化网络中任务相似度与遗忘呈单调关系
- 权重失衡促进弹性权重固化技术的性能提升
先前研究发现，在持续学习任务中，当任务间相似度适中时，灾难性遗忘最严重。这一现象通常归因于神经元在任务间复用已学特征。然而，该解释依赖于神经元专一性（即局部表示）存在的前提。我们通过理论框架揭示，专一性高度依赖初始条件：权重不平衡与高权重熵有利于产生专一解。在持续学习背景下，我们首先证明在非专一化网络中，任务相似度与遗忘之间存在单调关系；最后，表明通过权重失衡实现的专一性能有效提升常用弹性权重固化（Elastic Weight Consolidation, EWC）的效果。
原文摘要 · Abstract (English)
Prior work has demonstrated a consistent tendency in neural networks engaged in continual learning tasks, wherein intermediate task similarity results in the highest levels of catastrophic interference. This phenomenon is attributed to the network's tendency to reuse learned features across tasks. However, this explanation heavily relies on the premise that neuron specialisation occurs, i.e. the emergence of localised representations. Our investigation challenges the validity of this assumption. Using theoretical frameworks for the analysis of neural networks, we show a strong dependence of specialisation on the initial condition. More precisely, we show that weight imbalance and high weight entropy can favour specialised solutions. We then apply these insights in the context of continual learning, first showing the emergence of a monotonic relation between task-similarity and forgetting in non-specialised networks. {Finally, we show that specialization by weight imbalance is beneficial on the commonly employed elastic weight consolidation regularisation technique.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。