揭示深度学习泛化机制,解释过拟合为何不坏、训练如何加速。
A Theory of Generalization in Deep Learning

- 用神经正切核分解输出空间,信号通道快速积累,噪声通道误差被锁定
- 证明即使核变化大($oldsymbol{ ext{O}}(1)$),泛化仍成立,覆盖特征学习全阶段
- 提出可直接计算的噪声测量目标,无需验证集,提升训练效率与稳定性
我们提出一个非渐近的深度学习泛化理论,其中经验神经正切核将输出空间划分为不同方向。在对应信号的方向上,误差迅速消散;而在对应噪声的大量正交维度中,核的近零特征值将残余误差困于测试不可见的储库中。在信号通道内,小批量SGD使群体信号通过快速线性漂移累积,而个体记忆则被抑制为缓慢的扩散随机游走。我们证明,即使核在算子范数下变化$oldsymbol{ ext{O}}(1)$,泛化依然有效,涵盖完整的特征学习阶段。该理论自然解释了深度学习中的多种现象,如良性过拟合、双下降、隐式偏差和‘领悟’(grokking)。最后,我们从单次训练中推导出精确的种群风险目标,无需验证数据,适用于任意架构、损失或优化器,并证明其精准衡量信号通道中的噪声。该目标在实践中表现为对Adam的信噪比预处理,仅增加一个状态向量,成本为零;使‘领悟’加速5倍,抑制PINNs与隐式神经表示的记忆,同时在噪声偏好下改进DPO微调,且始终比参考策略更接近3倍。
原文摘要 · Abstract (English)
We present a non-asymptotic theory of generalization in deep learning where the empirical neural tangent kernel partitions the output space. In directions corresponding to signal, error dissipates rapidly; in the vast orthogonal dimensions corresponding to noise, the kernel's near-zero eigenvalues trap residual error in a test-invisible reservoir. Within the signal channel, minibatch SGD ensures that coherent population signal accumulates via fast linear drift, while idiosyncratic memorization is suppressed into a slow, diffusive random walk. We prove generalization survives even when the kernel evolves $\mathcal{O}(1)$ in operator norm, the full feature-learning regime. This theory naturally explains disparate phenomena in deep learning theory, such as benign overfitting, double descent, implicit bias, and grokking. Lastly, we derive an exact population-risk objective from a single training run with no validation data, for any architecture, loss, or optimizer, and prove that it measures precisely the noise in the signal channel. This objective reduces in practice to an SNR preconditioner on top of Adam, adding one state vector at no extra cost; it accelerates grokking by $5 \times$, suppresses memorization in PINNs and implicit neural representations, and improves DPO fine-tuning under noisy preferences while staying $3 \times$ closer to the reference policy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。