发现神经网络核与马特恩核在特定条件下预测高度一致
Correspondence of NNGP Kernel and the Matern Kernel
- 通过归一化构建有效NNGP核，解决数值稳定性问题
- NNGP核预测灵活性差，超参数变化影响小
- 在三个基准数据集上马特恩核表现更优，更适合实际应用
近年来，代表神经网络架构极限情况的核函数受到关注。然而，这些新核函数与现有核（如马特恩核）相比的应用效果尚不明确。本文采用实用方法研究神经网络高斯过程（NNGP）核在高斯过程回归中的应用。首先证明归一化对生成有效NNGP核的必要性，并探讨相关数值挑战。进一步发现该模型预测结果极不灵活，不同有效超参数下的输出差异很小。随后揭示一个意外现象：在特定条件下，NNGP核的预测结果与马特恩核极为接近，暗示过参数化深度神经网络与马特恩核之间存在深层相似性。最后，在三个基准数据集上对比了NNGP核与马特恩核的表现，结论是：由于灵活性和实际性能更优，马特恩核在实践中优于新型NNGP核。
原文摘要 · Abstract (English)
Kernels representing limiting cases of neural network architectures have recently gained popularity. However, the application and performance of these new kernels compared to existing options, such as the Matern kernel, is not well studied. We take a practical approach to explore the neural network Gaussian process (NNGP) kernel and its application to data in Gaussian process regression. We first demonstrate the necessity of normalization to produce valid NNGP kernels and explore related numerical challenges. We further demonstrate that the predictions from this model are quite inflexible, and therefore do not vary much over the valid hyperparameter sets. We then demonstrate a surprising result that the predictions given from the NNGP kernel correspond closely to those given by the Matern kernel under specific circumstances, which suggests a deep similarity between overparameterized deep neural networks and the Matern kernel. Finally, we demonstrate the performance of the NNGP kernel as compared to the Matern kernel on three benchmark data cases, and we conclude that for its flexibility and practical performance, the Matern kernel is preferred to the novel NNGP in practical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。