arXiv:2605.15435cs.LGcs.NE2026-05

新单元插入影响模型性能,需关注其训练稳定性。

On the Stability of Growth in Structural Plasticity

论文配图:On the Stability of Growth in Structural Plasticity
图 1 · 摘自论文原文
  • 新神经元插入后前向活跃但梯度弱,易被忽略。
  • 图像分类任务中生长法最终准确率高,但整体训练表现不如剪枝。
  • 新单元需足够时间整合,适合持续学习场景。

标准深度学习流程在训练前确定网络结构并保持不变。相比之下,可在训练过程中通过剪枝或增长调整结构。尽管生长看似是剪枝的逆过程,但二者本质不同:剪枝从已参与训练的单元中选择,而生长则将新单元插入已有优化轨迹。我们发现新生单元常表现为前向活跃但反向梯度信号弱,此差异在小型MLP中不明显,但在使用卷积主干的复杂图像分类任务中显著。实验显示,在结构编辑过程中, extsc{Grow} 可实现高最终准确率;而 extsc{Prune} 在训练轨迹平均性能或重新训练稀疏网络时表现更优。通过干预优化器状态、插入策略、选择机制和可训练性,发现提升新生单元融合能力可改善适应性表现,但不会自动提升最终子网性能。在强调结构可塑性损失的持续学习基准上, extsc{Grow} 仅在新生单元有充分整合时间时才具竞争力。结果表明, extsc{Grow} 应被视为依赖时间的优化过程,其成功取决于插入稳定性。

原文摘要 · Abstract (English)

Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization. In contrast, a model can also be adapted by editing its structure during training, for example by pruning existing hidden-neuron units or growing new ones. Although growth is appealing for adaptive and continual systems, we show that it is not simply the inverse of pruning. Pruning selects among units that have participated in training from the start, whereas growth inserts new units into an already specialized optimization trajectory. We isolate this insertion problem and show that newborn units are often forward-active but backward-starved: they participate in the forward computation, yet receive much weaker gradient signal than incumbent units. This disadvantage is minor in small MLP benchmarks, but becomes clear in harder image-classification settings with a convolutional trunk. In these settings, \textsc{Grow} can achieve high final accuracy during the structural-editing procedure, while \textsc{Prune} is stronger when performance is averaged over the training trajectory or when the final sparse network is retrained from scratch. Interventions targeting optimizer state, insertion, selection, and trainability show that improving the integration of newborn units can improve adaptive performance, but does not automatically produce better final subnetworks. In continual-learning benchmarks stressing plasticity loss, \textsc{Grow} becomes competitive mainly when new units have enough time to integrate. Together, these results suggest that \textsc{Grow} should be evaluated not only as an architecture-search operator, but as a time-sensitive optimization process whose success depends on insertion stability.

结构可塑性持续学习神经网络生长梯度稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。