揭示深度ReLU网络宽度与深度对记忆能力的权衡关系。
Memorization capacity of deep ReLU neural networks characterized by width and depth
- 通过构造满足W²L²=O(N log(δ⁻¹))的网络实现任意数据记忆
- 证明下界W²L²=Ω(N log(δ⁻¹))，构建最优（对数因子内）
- 适用于研究网络容量与泛化关系的研究者
本文研究带ReLU激活函数的深度神经网络的记忆容量。具体而言，我们考察在单位球内任意N个数据点（两两间距离至少为δ）和离散标签的情况下，最小规模的神经网络应具备何种宽度W与深度L。以往研究多以参数量或神经元数刻画记忆能力。本文通过构造满足W²L² = O(N log(δ⁻¹))的网络，可实现任意N个样本的记忆；同时证明任何此类网络必须满足下界W²L² = Ω(N log(δ⁻¹))。该结果表明，在δ⁻¹关于N为多项式时，构造是近乎最优的，从而显式刻画了深度-宽度对记忆容量的权衡关系。
原文摘要 · Abstract (English)
This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $δ$ and discrete labels. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We generalize these results by constructing neural networks, whose width $W$ and depth $L$ satisfy $W^2L^2= \mathcal{O}(N\log(δ^{-1}))$, that can memorize any $N$ data samples. We also prove that any such networks should also satisfy the lower bound $W^2L^2=Ω(N \log(δ^{-1}))$, which implies that our construction is optimal up to logarithmic factors when $δ^{-1}$ is polynomial in $N$. Hence, we explicitly characterize the trade-off between width and depth for the memorization capacity of deep neural networks in this regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。