研究重尾权重下神经网络核矩阵的谱分布，发现其具有独特且更复杂的特征值行为。
Global law of conjugate kernel random matrices with heavy-tailed weights
- 基于重尾权重与轻尾输入构建两层神经网络核矩阵
- 推导出特征值分布极限，揭示重尾导致强相关性与新谱态
- 适用于分析深度学习中非正则权重对模型行为的影响
我们研究了两层神经网络模型中协变核随机矩阵 $YY^ op$ 的渐近谱分布，其中 $Y = f(WX)$。$W$ 和 $X$ 为独立同分布的随机矩形矩阵，$W$ 的元素服从重尾分布（如对称 α-稳定分布，α ∈ (0,2)），而 $X$ 的元素为轻尾分布。激活函数 $f$ 为有界、光滑、奇函数且非线性。通过计算矩，我们得到了 $YY^ op$ 的极限特征值分布，表明重尾权重会引发 $Y$ 元素间的强相关性，导致其谱行为显著区别于轻尾情况。
原文摘要 · Abstract (English)
We study the asymptotic spectral distribution of the conjugate kernel random matrix $YY^\top$, where $Y= f(WX)$ arises from a two-layer neural network model. We consider the setting where $W$ and $X$ are random rectangular matrices with i.i.d.\ entries, where the entries of $W$ follow a heavy-tailed distribution, while those of $X$ have light tails. Our assumptions on $W$ include a broad class of heavy-tailed distributions, such as symmetric $α$-stable laws with $α\in ]0,2[$ and sparse matrices with $\mathcal{O}(1)$ nonzero entries per row. The activation function $f$, applied entrywise, is bounded, smooth, odd, and nonlinear. We compute the limiting eigenvalue distribution of $YY^\top$ through its moments and show that heavy-tailed weights induce strong correlations between the entries of $Y$, resulting in richer and fundamentally different spectral behavior compared to the light-tailed case.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。