研究深度神经网络在混沌边缘的核函数，发现其谱性质随深度提升而更稳定。
MLPs at the EOC: Spectrum of the NTK
- 用逆余弦距离近似无限宽多层感知机的神经正切核
- 证明核矩阵条件数收敛速度由激活函数参数决定，越深越稳定
- 对混沌边缘初始化的模型提供紧致谱界，适合研究深度网络泛化性
我们研究了在混沌边缘（EOC）初始化的无限宽$l$层多层感知机（MLP）对应的神经正切核（NTK）$ ilde{K} : bR^{m_0} imes bR^{m_0} o bR^{m_l imes m_l}$ 的性质。该类激活函数形式为 $ϕ(s) = a s + b |s|$。随着深度$l$增加，核矩阵元素$ ilde{K}(x_1,x_2)$可被对应输入激活值间余弦距离的倒数越来越精确地逼近。通过量化这些逆余弦距离及其构成矩阵的谱特性，我们获得了在数据集$\\[x_1,\\cdots,x_n\ ] \subset \bbR^{m_0}$上NTK矩阵$ ilde{K} = [\frac{1}{n} \tilde{K}(x_{i_1},x_{i_2}) : i_1, i_2 \in [1:n]]$的紧致谱界。结果表明，参数比值$Δ_ϕ = \frac{b^2}{a^2+b^2}$决定了核矩阵条件数向极限收敛的速度；当$Δ_ϕ=1$时（绝对值函数），收敛速度优于ReLU（$Δ_ϕ=\frac{1}{2}$）。
原文摘要 · Abstract (English)
We study the properties of the Neural Tangent Kernel (NTK) $\overset{\scriptscriptstyle\infty}{K} : \mathbb{R}^{m_0} \times \mathbb{R}^{m_0} \to \mathbb{R}^{m_l \times m_l}$ corresponding to infinitely wide $l$-layer Multilayer Perceptrons (MLPs) taking inputs from $\mathbb{R}^{m_0}$ to outputs in $\mathbb{R}^{m_l}$ equipped with activation functions $ϕ(s) = a s + b \vert s \vert$ for some $a,b \in \mathbb{R}$ and initialized at the Edge Of Chaos (EOC). We find that the entries $\overset{\scriptscriptstyle\infty}{K}(x_1,x_2)$ can be approximated by the inverses of the cosine distances of the activations corresponding to $x_1$ and $x_2$ increasingly better as the depth $l$ increases. By quantifying these inverse cosine distances and the spectrum of the matrix containing them, we obtain tight spectral bounds for the NTK matrix $\overset{\scriptscriptstyle\infty}{K} = [\frac{1}{n} \overset{\scriptscriptstyle\infty}{K}(x_{i_1},x_{i_2}) : i_1, i_2 \in [1:n]]$ over a dataset $\{x_1,\cdots,x_n\} \subset \mathbb{R}^{m_0}$, transferred from the inverse cosine distance matrix via our approximation result. Our results show that $Δ_ϕ= \frac{b^2}{a^2+b^2}$ determines the rate at which the condition number of the NTK matrix converges to its limit as depth increases, implying in particular that the absolute value ($Δ_ϕ=1$) is better than the ReLU ($Δ_ϕ=\frac{1}{2}$) in this regard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。