揭示神经网络初始化时的有效秩恒定性，解释为何大模型仍能良好泛化。
The Spectral Dimension of NTKs is Constant: A Theory of Implicit Regularization, Finite-Width Stability, and Scalable Estimation
- 通过分析NTK核矩阵的迹与范数比，定义有效秩衡量初始复杂度。
- 在无限宽极限下，有效秩趋于常数1.0–1.3，且随样本数变化极小。
- 提出可扩展估计方法，适用于实际大模型，结果与理论预测高度一致。
现代深度网络参数量巨大却常能良好泛化，暗示其内在复杂度远低于参数数量。本文通过分析初始化时神经正切核（NTK）Gram矩阵的有效秩 $r_{\text{eff}}(K) = (\text{tr}(K))^2/\|K\|_F^2$ 来研究这种复杂度。对于独立同分布数据和无限宽NTK核 $k$，我们证明了 $\lim_{n\to\infty} \mathbb{E}[r_{\text{eff}}(K_n)] = \mathbb{E}[k(x, x)]^2 / \mathbb{E}[k(x, x')^2] =: r_\infty$，且具有次高斯浓度性质。进一步建立了有限宽度稳定性：若有限宽NTK在算子范数上偏离 $O_p(m^{-1/2})$（宽度 $m$），则 $r_{\text{eff}}$ 变化也为 $O_p(m^{-1/2})$。设计了一种基于随机输出探针与参数雅可比矩阵的CountSketch的可扩展估计器，证明其条件无偏性和一致性，并给出显式方差界。在CIFAR-10上使用ResNet-20/56（宽度16/32）对 $n \in \{10^3, 5\times10^3, 10^4, 2.5\times10^4, 5\times10^4\}$ 进行实验，观测到 $r_{\text{eff}} \approx 1.0\text{--}1.3$，且随 $n$ 变化的斜率接近0，符合理论预期，核矩预测值与拟合常数吻合良好。
原文摘要 · Abstract (English)
Modern deep networks are heavily overparameterized yet often generalize well, suggesting a form of low intrinsic complexity not reflected by parameter counts. We study this complexity at initialization through the effective rank of the Neural Tangent Kernel (NTK) Gram matrix, $r_{\text{eff}}(K) = (\text{tr}(K))^2/\|K\|_F^2$. For i.i.d. data and the infinite-width NTK $k$, we prove a constant-limit law $\lim_{n\to\infty} \mathbb{E}[r_{\text{eff}}(K_n)] = \mathbb{E}[k(x, x)]^2 / \mathbb{E}[k(x, x')^2] =: r_\infty$, with sub-Gaussian concentration. We further establish finite-width stability: if the finite-width NTK deviates in operator norm by $O_p(m^{-1/2})$ (width $m$), then $r_{\text{eff}}$ changes by $O_p(m^{-1/2})$. We design a scalable estimator using random output probes and a CountSketch of parameter Jacobians and prove conditional unbiasedness and consistency with explicit variance bounds. On CIFAR-10 with ResNet-20/56 (widths 16/32) across $n \in \{10^3, 5\times10^3, 10^4, 2.5\times10^4, 5\times10^4\}$, we observe $r_{\text{eff}} \approx 1.0\text{--}1.3$ and slopes $\approx 0$ in $n$, consistent with the theory, and the kernel-moment prediction closely matches fitted constants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。