arXiv:2603.12316cond-mat.dis-nncs.LG2026-03被引 1

用剪枝研究神经网络相变,发现三类学习状态。

Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

  • 通过控制训练和推理时的丢弃率,绘制全连接网络相图。
  • 损失随数据量呈幂律衰减,三类相:能学、忘光、学不会。
  • 相变具尺度不变性,类似玻色-库珀-索利斯相变,适合物理背景研究者。

现代神经网络高度过参数化,剪枝通过移除冗余神经元或连接成为压缩模型的重要手段,且不牺牲性能。然而,尽管实用剪枝方法成熟,剪枝是否在神经网络中引发突变相变及其所属普适类仍属未知。为此,我们研究了在MNIST上训练的全连接神经网络,独立调节训练与评估阶段的丢弃率以绘制相图。识别出三个显著相态:eumentia(网络可学习)、dementia(网络已遗忘)和amentia(网络无法学习),其区分依据是交叉熵损失随训练集规模的幂律缩放特征。在eumentia相中,损失的代数衰减——机器学习文献中记载的神经网络缩放定律——从统计力学视角看,标志着准长程有序。我们证明,eumentia与dementia相之间的转变伴随尺度不变性,具有发散长度标度,展现出类贝雷津斯基-科斯特里茨-索利斯相变特征;该相结构对不同网络宽度与深度均保持鲁棒。结果表明,丢弃诱导的剪枝提供了一个可借助统计力学理解神经网络行为的具体框架。

原文摘要 · Abstract (English)

Modern neural networks are heavily overparameterized, and pruning, which removes redundant neurons or connections, has emerged as a key approach to compressing them without sacrificing performance. However, while practical pruning methods are well developed, whether pruning induces sharp phase transitions in the neural networks and, if so, to what universality class they belong, remain open questions. To address this, we study fully-connected neural networks trained on MNIST, independently varying the dropout (i.e., removing neurons) rate at both the training and evaluation stages to map the phase diagram. We identify three distinct phases: eumentia (the network learns), dementia (the network has forgotten), and amentia (the network cannot learn), sharply distinguished by the power-law scaling of the cross-entropy loss with the training dataset size. {In the eumentia phase, the algebraic decay of the loss, as documented in the machine learning literature as neural scaling laws, is from the perspective of statistical mechanics the hallmark of quasi-long-range order.} We demonstrate that the transition between the eumentia and dementia phases is accompanied by scale invariance, with a diverging length scale that exhibits hallmarks of a Berezinskii-Kosterlitz-Thouless-like transition; the phase structure is robust across different network widths and depths. Our results establish that dropout-induced pruning provides a concrete setting in which neural network behavior can be understood through the lens of statistical mechanics.

神经网络相变剪枝统计力学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。