发现ReLU模型在原点附近会呈现二次外推，突破了传统线性认知。
A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernel
- 在神经正切核框架下研究原点附近的外推行为
- 证明原点附近存在二次外推而非线性，与远点不同
- 为理解深度网络外推机制提供新视角，适合理论研究者
已有理论和实证表明，ReLU MLP 在分布外评估点上倾向于线性外推。尽管机器学习文献已广泛分析线性外推的机制，但神经正切核（NTK）框架下原点附近的外推分析仍是一个未充分探索的特殊情形。特别是，由神经正切核诱导的无限维特征映射不具备平移不变性，这意味着远离原点的分布外点与靠近原点的点的外推行为并不等价。由于该特征映射具有旋转不变性，这两类情况可能代表了ReLU NTK外推的最典型极端边界。正是对这两种特殊外推情形的模糊认识，促使本文发现了靠近原点时的二次外推现象。
原文摘要 · Abstract (English)
It has been demonstrated both theoretically and empirically that the ReLU MLP tends to extrapolate linearly for an out-of-distribution evaluation point. The machine learning literature provides ample analysis with respect to the mechanisms to which linearity is induced. However, the analysis of extrapolation at the origin under the NTK regime remains a more unexplored special case. In particular, the infinite-dimensional feature map induced by the neural tangent kernel is not translationally invariant. This means that the study of an out-of-distribution evaluation point very far from the origin is not equivalent to the evaluation of a point very near the origin. And since the feature map is rotation invariant, these two special cases may represent the most canonically extreme bounds of ReLU NTK extrapolation. Ultimately, it is this loose recognition of the two special cases of extrapolation that motivate the discovery of quadratic extrapolation for an evaluation close to the origin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。