arXiv:2603.02234cs.LGcs.AI2026-03

对比结构化与非结构化剪枝,发现前者效率低得多。

Structured vs. Unstructured Pruning: An Exponential Gap

  • 用两层ReLU网络剪掉隐藏单元逼近单个神经元
  • 结构化剪枝需Ω(1/ε)规模网络,非结构化仅需O(log(1/ε))
  • 揭示剪枝方法本质差异,适合关注硬件部署的读者

强彩票券假说(SLTH)认为大型随机初始化神经网络中存在稀疏子网络,可在不训练的情况下近似目标函数,表明剪枝本身已足够。现有理论多支持非结构化剪枝,证明对数级过参数化即可逼近简单目标网络;而结构化剪枝(如神经元剪枝)虽具实用价值,但缺乏理论支持。本文研究通过剪除两层ReLU网络的隐藏单元来逼近单个无偏置的ReLU神经元,揭示其内在局限。结果表明:实现ε-近似需Ω(1/ε)规模起始网络,而非结构化剪枝仅需O(log(1/ε))个隐藏单元,两者间存在指数级差距。

原文摘要 · Abstract (English)

The Strong Lottery Ticket Hypothesis (SLTH) states that large, randomly initialized neural networks contain sparse subnetworks capable of approximating a target function at initialization without training, suggesting that pruning alone is sufficient. Pruning methods are typically classified as unstructured, where individual weights can be removed from the network, and structured, where parameters are removed according to specific patterns, as in neuron pruning. Existing theoretical results supporting the SLTH rely almost exclusively on unstructured pruning, showing that logarithmic overparameterization suffices to approximate simple target networks. In contrast, neuron pruning has received limited theoretical attention, despite its practical appeal for direct hardware speedups. In this work, we consider the problem of approximating a single bias-free ReLU neuron by pruning hidden units of a randomly initialized two-layer ReLU network, effectively isolating the intrinsic limitations of neuron pruning. We show that achieving an $\varepsilon$-approximation requires a starting network size of $Ω(1/\varepsilon)$ for neuron pruning, whereas weight pruning succeeds with only $O(\log(1/\varepsilon))$ hidden units, revealing an exponential separation between the two approaches.

剪枝理论分析神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。