让神经网络训练时动态长大，比一开始就大更高效。
Growing Neural Networks: Dynamic Evolution through Gradient Descent
- 用辅助权重或控制器掩码控制网络大小，随训练生长。
- 在回归与分类任务中，动态成长网络优于同规模静态网络。
- 从小开始自然生长，适合追求能效与可扩展性的研究者。
与传统结构固定的神经网络不同，本文提出两种在训练过程中使小型网络动态演变为更大网络的方法。第一种通过辅助权重直接控制网络规模，第二种则利用控制器生成的掩码调节神经元参与度。两种方法均通过与权重和偏置相同的梯度下降算法优化网络规模。我们在非线性回归和分类任务上评估这些可生长网络，结果表明其始终优于同等最终规模的静态网络。进一步探索了网络超参数空间，发现其与静态网络存在特定缩放关系。结果表明，从小开始并自然生长，可能比直接设定大规模更优，尤其当神经网络持续增大且能耗增加时。
原文摘要 · Abstract (English)
In contrast to conventional artificial neural networks, which are structurally static, we present two approaches for evolving small networks into larger ones during training. The first method employs an auxiliary weight that directly controls network size, while the second uses a controller-generated mask to modulate neuron participation. Both approaches optimize network size through the same gradient-descent algorithm that updates the network's weights and biases. We evaluate these growing networks on nonlinear regression and classification tasks, where they consistently outperform static networks of equivalent final size. We then explore the hyperparameter space of these networks to find associated scaling relations relative to their static counterparts. Our results suggest that starting small and growing naturally may be preferable to simply starting large, particularly as neural networks continue to grow in size and energy consumption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。