研究 MLP 在混沌边缘的 NTK 浓度，发现有限宽度下也能逼近无限宽极限。
MLPs at the EOC: Concentration of the NTK
- 在混沌边缘初始化，不依赖渐近假设证明梯度近似独立
- 证明 NTK 矩阵在有限宽度下集中于无限宽极限，无需线性过参数化
- 指出绝对值激活函数比 ReLU 更利于 NTK 集中，宽度需平方增长
本文研究了在边沿混沌（EOC）初始化下的 l 层 MLP 模型的神经正切核（NTK）浓度性质。针对形如 ϕ(s) = a s + b |s| 的激活函数，作者在不依赖梯度独立性假设（该假设仅在无限宽极限下成立）的前提下，证明了有限宽度下梯度近似独立性仍成立。通过最大不等式，证明了在数据集 {x₁,…,xₙ} 上所有 NTK 项 K_θ(x_{i₁},x_{i₂}) 能同时集中。进一步得出，归一化 NTK 矩阵 K(θ) ∈ ℝ^{nm_l×nm_l} 能集中在无限宽极限矩阵 Ķ ∈ ℝ^{nm_l×nm_l}，且无需线性过参数化。结果表明，为实现充分集中，隐层宽度需满足 m_k = k²m（m ∈ ℕ⁺）。此时，浓度界为 ℙ(‖K(θ) - Ķ‖ ≤ O((Δ_ϕ⁻² + m_l^{1/2}l) κ_ϕ² m^{-1/2})) ≥ 1 - O(m⁻¹)，其中 Δ_ϕ = b²/(a²+b²)，κ_ϕ = (|a|+|b|)/√(a²+b²)。分析显示，绝对值激活函数（Δ_ϕ=1, κ_ϕ=1）优于 ReLU（Δ_ϕ=1/2, κ_ϕ=√2）。
原文摘要 · Abstract (English)
We study the concentration of the Neural Tangent Kernel (NTK) $K_θ: \mathbb{R}^{m_0} \times \mathbb{R}^{m_0} \to \mathbb{R}^{m_l \times m_l}$ of $l$-layer Multilayer Perceptrons (MLPs) $N : \mathbb{R}^{m_0} \times Θ\to \mathbb{R}^{m_l}$ equipped with activation functions $ϕ(s) = a s + b \vert s \vert$ for some $a,b \in \mathbb{R}$ with the parameter $θ\in Θ$ being initialized at the Edge Of Chaos (EOC). Without relying on the gradient independence assumption that has only been shown to hold asymptotically in the infinitely wide limit, we prove that an approximate version of gradient independence holds at finite width. Showing that the NTK entries $K_θ(x_{i_1},x_{i_2})$ for $i_1,i_2 \in [1:n]$ over a dataset $\{x_1,\cdots,x_n\} \subset \mathbb{R}^{m_0}$ concentrate simultaneously via maximal inequalities, we prove that the NTK matrix $K(θ) = [\frac{1}{n} K_θ(x_{i_1},x_{i_2}) : i_1,i_2 \in [1:n]] \in \mathbb{R}^{nm_l \times nm_l}$ concentrates around its infinitely wide limit $\overset{\scriptscriptstyle\infty}{K} \in \mathbb{R}^{nm_l \times nm_l}$ without the need for linear overparameterization. Our results imply that in order to accurately approximate the limit, hidden layer widths have to grow quadratically as $m_k = k^2 m$ for some $m \in \mathbb{N}+1$ for sufficient concentration. For such MLPs, we obtain the concentration bound $\mathbb{P}( \Vert K(θ) - \overset{\scriptscriptstyle\infty}{K} \Vert \leq O((Δ_ϕ^{-2} + m_l^{\frac{1}{2}} l) κ_ϕ^2 m^{-\frac{1}{2}})) \geq 1-O(m^{-1})$ modulo logarithmic terms, where we denoted $Δ_ϕ= \frac{b^2}{a^2+b^2}$ and $κ_ϕ= \frac{\vert a \vert + \vert b \vert}{\sqrt{a^2 + b^2}}$. This reveals in particular that the absolute value ($Δ_ϕ=1$, $κ_ϕ=1$) beats the ReLU ($Δ_ϕ=\frac{1}{2}$, $κ_ϕ=\sqrt{2}$) in terms of the concentration of the NTK.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。