证明强彩票票假设中子网络可稀疏,且首次给出稀疏性保证。
On the Sparsity of the Strong Lottery Ticket Hypothesis
- 通过改进随机固定大小子集和问题,构建稀疏子网络
- 在稠密与等变网络中实现稀疏子网络的准确近似
- 为彩票票假设提供理论支撑,适合关注神经网络压缩的研究者
近期研究试图证明:一个随机神经网络 $N$ 中存在子网络,可在不训练的情况下精确逼近任何比 $N$ 小得多的目标网络。这一方向被称为强彩票票假设(SLTH),最初源于较弱版本的彩票票假设,后者指出大型随机网络中存在稀疏子网络,经训练后性能可媲美整个网络。然而,现有成果未对子网络大小提供任何保证,根源在于其依赖的随机子集和(RSS)问题的性质。本文针对经典架构(如稠密与等变网络),首次在理论上证明了 SLTH,并给出了子网络稀疏性的明确保障。核心贡献是证明了随机固定大小子集和问题(RFSS)的近乎紧致界,该问题作为 RSS 的变体具有独立意义。
原文摘要 · Abstract (English)
Considerable research efforts have recently been made to show that a random neural network $N$ contains subnetworks capable of accurately approximating any given neural network that is sufficiently smaller than $N$, without any training. This line of research, known as the Strong Lottery Ticket Hypothesis (SLTH), was originally motivated by the weaker Lottery Ticket Hypothesis, which states that a sufficiently large random neural network $N$ contains \emph{sparse} subnetworks that can be trained efficiently to achieve performance comparable to that of training the entire network $N$. Despite its original motivation, results on the SLTH have so far not provided any guarantee on the size of subnetworks. Such limitation is due to the nature of the main technical tool leveraged by these results, the Random Subset Sum (RSS) Problem. Informally, the RSS Problem asks how large a random i.i.d. sample $Ω$ should be so that we are able to approximate any number in $[-1,1]$, up to an error of $ ε$, as the sum of a suitable subset of $Ω$. We provide the first proof of the SLTH in classical settings, such as dense and equivariant networks, with guarantees on the sparsity of the subnetworks. Central to our results, is the proof of an essentially tight bound on the Random Fixed-Size Subset Sum Problem (RFSS), a variant of the RSS Problem in which we only ask for subsets of a given size, which is of independent interest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。