arXiv:2409.01568cs.LG2024-09被引 2

通过量化网络涌现性,揭示其与训练性能和复杂度的关系。

Quantifying Emergence in Neural Networks: Insights from Pruning and Training Dynamics

  • 用节点活跃性连接度衡量网络涌现性。
  • 高涌现性对应更好训练效果和更复杂的损失曲面。
  • 剪枝提升训练效率但可能降低最终准确率。

涌现性指网络中简单单元相互作用产生复杂行为的现象,在提升神经网络能力方面起关键作用。本文提出一种量化框架,用于测量训练过程中的涌现性,并研究其对网络性能的影响,尤其关注剪枝与训练动态的关系。假设指出,由活跃与非活跃节点间连通性定义的涌现程度,可预测网络中涌现行为的发展。在基准数据集上的前馈与卷积架构实验表明,更高的涌现性与更好的可训练性和性能相关。进一步分析显示,网络复杂度与损失景观存在关联,高涌现性意味着更多局部极小值聚集,损失曲面更崎岖。剪枝通过移除冗余节点与连接降低网络复杂度,可提升训练效率与收敛速度,但可能导致最终准确率下降。这些发现为理解涌现性、复杂度与性能之间的关系提供了新视角,对高效架构的设计与优化具有重要启示。

原文摘要 · Abstract (English)

Emergence, where complex behaviors develop from the interactions of simpler components within a network, plays a crucial role in enhancing neural network capabilities. We introduce a quantitative framework to measure emergence during the training process and examine its impact on network performance, particularly in relation to pruning and training dynamics. Our hypothesis posits that the degree of emergence, defined by the connectivity between active and inactive nodes, can predict the development of emergent behaviors in the network. Through experiments with feedforward and convolutional architectures on benchmark datasets, we demonstrate that higher emergence correlates with improved trainability and performance. We further explore the relationship between network complexity and the loss landscape, suggesting that higher emergence indicates a greater concentration of local minima and a more rugged loss landscape. Pruning, which reduces network complexity by removing redundant nodes and connections, is shown to enhance training efficiency and convergence speed, though it may lead to a reduction in final accuracy. These findings provide new insights into the interplay between emergence, complexity, and performance in neural networks, offering valuable implications for the design and optimization of more efficient architectures.

神经网络涌现性剪枝训练动态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。