NTK理论声称神经网络等价于核回归，但实验证明二者实际表现差异显著。
Issues with Neural Tangent Kernel Approach to Neural Networks
- 重新推导NTK并进行数值实验验证其等价性
- 增加网络层后预测误差变化不匹配，暴露理论缺陷
- 普通高斯核的预测误差与NTK几乎相同，说明NTK未必反映真实训练过程
神经切线核（NTK）被提出用于从高斯过程视角研究训练后神经网络的行为。该领域一个重要成果是证明训练后的神经网络与对应NTK的核回归等价。这一结论使神经网络可被理解为核回归的特例。然而，这种等价性在实践中是否成立？本文严谨重审了NTK的推导过程，并通过数值实验评估该等价定理。结果发现：增加一个网络层及其对应的更新后NTK，无法带来预测误差的同步变化。此外，文献中未考虑神经网络训练的高斯过程核所对应的核回归，其预测误差与使用NTK的核回归非常接近。这些观察表明，等价定理在实践中并不成立，质疑了NTK是否真正刻画了神经网络的训练过程。
原文摘要 · Abstract (English)
Neural tangent kernels (NTKs) have been proposed to study the behavior of trained neural networks from the perspective of Gaussian processes. An important result in this body of work is the theorem of equivalence between a trained neural network and kernel regression with the corresponding NTK. This theorem allows for an interpretation of neural networks as special cases of kernel regression. However, does this theorem of equivalence hold in practice? In this paper, we revisit the derivation of the NTK rigorously and conduct numerical experiments to evaluate this equivalence theorem. We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error. Furthermore, we observe that kernel regression with a Gaussian process kernel in the literature that does not account for neural network training produces prediction errors very close to that of kernel regression with NTKs. These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。