新添神经元易失效？该方法让它们高效协同工作。
Shared-Weights Extender and Gradient Voting for Neural Network Expansion
- 通过共享权重机制连接新旧神经元，促进平滑融合
- 在四个数据集上抑制神经元失活率，性能超越基线
- 适合需要动态扩展模型容量的场景，如持续学习
训练中扩展神经网络是一种提升模型容量而无需从头训练大模型的有前景方法。然而，新增神经元常因无法适应已训练网络而陷入无效状态，无法贡献容量增长。本文提出共享权重扩展器（SWE），一种专为防止新神经元失效而设计的新方法，通过将新旧神经元耦合实现平滑集成。同时引入梯度投票分配器（SVoD），基于梯度分配各层新增神经元。在四个数据集上的广泛实验表明，该方法能有效抑制神经元失活，并优于其他扩展方法与基线模型。
原文摘要 · Abstract (English)
Expanding neural networks during training is a promising way to augment capacity without retraining larger models from scratch. However, newly added neurons often fail to adjust to a trained network and become inactive, providing no contribution to capacity growth. We propose the Shared-Weights Extender (SWE), a novel method explicitly designed to prevent inactivity of new neurons by coupling them with existing ones for smooth integration. In parallel, we introduce the Steepest Voting Distributor (SVoD), a gradient-based method for allocating neurons across layers during deep network expansion. Our extensive benchmarking on four datasets shows that our method can effectively suppress neuron inactivity and achieve better performance compared to other expanding methods and baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。