深度网络学习动态由相关性传播决定,临界初始化是关键。
Correlation flow governs learning at criticality

- 用平均场与随机矩阵理论建立相关性传播与学习核的联系。
- 仅在权重-偏置方差平面的临界点,信息可无限深层传播。
- 正交初始化能抑制有限尺寸偏差,适合深网训练研究。
深度神经网络的初始化决定了信息与梯度能否跨层传播,但其与学习动态的统一理论仍不明确。结合平均场理论与随机矩阵理论,我们建立了相关性传播与控制序列极限下无限宽、无限深网络学习的神经正切核(NTK)之间的直接关联。相关性传播至无限深度仅在权重-偏置方差平面上的单一临界点实现。在此点,利用端到端雅可比矩阵随深度呈代数衰减的特性,证明了NTK恰好与无限深度下的输出相关性成正比,从而将信息传播与学习动态统一。进一步表明,正交初始化可抑制高斯初始化中存在的主导有限尺寸修正项,阐明了两种初始化在该极限下的作用差异。这些理论预测在有限宽度、有限深度网络上得到定量验证。结果共同表明,正交初始化与临界性是控制深度学习渐近动态的必要条件。
原文摘要 · Abstract (English)
The initialization of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single, critical point in the weight-bias variance plane. At this point, we leverage the algebraic decay of the end-to-end Jacobian with depth to prove that the NTK becomes exactly proportional to the output correlation at infinite depth, tying together information propagation and learning dynamics. We further show that orthogonal initialization suppresses the leading finite-size corrections present under Gaussian initialization, clarifying the respective roles of the two initialization ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialization and criticality are required to control the asymptotic dynamics of deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。