揭示有限宽度ReLU网络中逐层有效维度的振荡规律。
Some Theoretical Results on Layerwise Effective Dimension Oscillations in Finite Width ReLU Networks
- 推导出随机高斯权重下隐藏层特征矩阵的期望秩公式
- 发现秩缺陷按约0.3634的比率几何衰减，存在局部峰值
- 该振荡是有限宽度特有现象，适用于理解深度网络表达能力
我们分析了有限宽度全连接ReLU网络中逐层有效维度（特征矩阵的秩）。针对固定批量m个输入和随机高斯权重，推导出m×n隐藏激活矩阵的期望秩的闭式表达式。主要结果表明，E[EDim(ℓ)] = m[1−(1−2/π)ℓ]+O(e−cm)，秩缺陷以约0.3634的比率几何衰减。还证明了次高斯浓度界，并识别出期望秩达到局部极大值的“复兴”深度，其位置ℓ*_k ≈ (k+1/2)π/log(1/ρ)，高度约为0.79m。进一步表明，这种振荡行为是有限宽度的特有现象：在正交初始化或强负斜率Leaky-ReLU下，秩保持近乎满秩。这些结果精确刻画了随机ReLU层对输入变化子空间的交替坍缩与部分恢复机制，为深度网络表达能力研究提供了新视角。
原文摘要 · Abstract (English)
We analyze the layerwise effective dimension (rank of the feature matrix) in fully-connected ReLU networks of finite width. Specifically, for a fixed batch of $m$ inputs and random Gaussian weights, we derive closed-form expressions for the expected rank of the \$m\times n\$ hidden activation matrices. Our main result shows that $\mathbb{E}[EDim(\ell)]=m[1-(1-2/π)^\ell]+O(e^{-c m})$ so that the rank deficit decays geometrically with ratio $1-2 / π\approx 0.3634$. We also prove a sub-Gaussian concentration bound, and identify the "revival" depths at which the expected rank attains local maxima. In particular, these peaks occur at depths $\ell_k^*\approx(k+1/2)π/\log(1/ρ)$ with height $\approx (1-e^{-π/2}) m \approx 0.79m$. We further show that this oscillatory rank behavior is a finite-width phenomenon: under orthogonal weight initialization or strong negative-slope leaky-ReLU, the rank remains (nearly) full. These results provide a precise characterization of how random ReLU layers alternately collapse and partially revive the subspace of input variations, adding nuance to prior work on expressivity of deep networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。