首次证明了稀疏ReLU网络的精确恢复理论，适用于两层输出网络。
A Recovery Guarantee for Sparse Neural Networks
- 采用迭代硬阈值算法，线性内存实现稀疏权重精确恢复。
- 实验显示在植网、手写数字分类和隐式神经表示上性能优越。
- 适合关注稀疏网络理论保证与高效训练的研究者。
我们首次为ReLU神经网络提供了稀疏恢复的理论保证，其中稀疏网络权重即为待恢复信号。针对两层、标量输出网络，研究了稀疏权重的结构特性，证明简单迭代硬阈值算法可在权重非零数线性增长的内存下精确恢复这些权重。通过简单实验验证了该理论结果，涵盖稀疏植网恢复、MNIST分类以及隐式神经表示。实验表明，该方法性能可媲美甚至超过基于迭代幅度剪枝的高性能但内存低效基线方法。代码已公开于https://github.com/voilalab/MLP-IHT。
原文摘要 · Abstract (English)
We prove the first guarantees of sparse recovery for ReLU neural networks, where the sparse network weights constitute the signal to be recovered. Specifically, we study structural properties of the sparse network weights for two-layer, scalar-output networks under which a simple iterative hard thresholding algorithm recovers these weights exactly, using memory that grows linearly in the number of nonzero weights. We validate this theoretical result with simple experiments on recovery of sparse planted MLPs, MNIST classification, and implicit neural representations. Experimentally, we find performance that is competitive with, and often exceeds, a high-performing but memory-inefficient baseline based on iterative magnitude pruning. Code is available at https://github.com/voilalab/MLP-IHT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。