揭示神经网络预测的二阶路径核机制，深化对优化过程的理解。
Second-Order Path Kernel Interpolation Formulas in Machine Learning
- 提出二阶路径核插值公式，引入曲率加权项提升预测精度
- 发现随机梯度下降中噪声协方差与预测曲率的耦合效应
- 适用于带动量的优化方法，且给出预测波动范围的估计
理解训练数据如何影响神经网络预测是现代学习理论的核心问题。2020年，Pedro Domingos 提出一个适用于确定性梯度下降模型的插值公式，将模型预测表示为沿优化路径的积分，其权重由数据相关核决定，该核对齐测试与训练数据处的梯度。这一阶特征在基于批次的随机优化中依然成立。本文发展了二阶形式的插值公式：主路径核插值项外，新增曲率加权项；对于随机梯度下降，还出现由采样引起的额外成分，该成分将预测曲率与小批量梯度噪声的协方差相耦合。我们还将该框架扩展至带动量的随机梯度下降，保留插值结构但权重受记忆因子修正。此外，我们建立了终端预测的集中估计，明确了围绕预期二阶表示的波动尺度。这些结果共同实现了对神经网络预测路径核解释的精细化。
原文摘要 · Abstract (English)
Understanding how training data shape neural network predictions is a central problem in modern learning theory. In 2020, Pedro Domingos proposed an interpolation formula valid for every model learned by deterministic gradient descent. It expresses the model's prediction as an integral, along the optimization path, of a data-dependent kernel that aligns the model's gradients at the test and training data. Such a first-order characterization remains valid for models trained with batch-based stochastic optimization. In this paper, we develop second-order forms of these interpolation formulas. We show that the leading path-kernel interpolation is supplemented by a curvature-weighted interpolation term. For stochastic gradient descent, an additional sampling-induced component appears, coupling the curvature of the prediction with the covariance of mini-batch gradient noise. We also extend the representation to stochastic gradient descent with momentum, where the interpolation structure is preserved but with the weights modified by a memory-related factor. Moreover, we establish a concentration estimate for the terminal prediction, identifying the fluctuation scale around the expected second-order representation. Together, these results provide a refinement of the path-kernel interpretation of neural network prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。