模型越宽，可剪枝比例却不变，说明大模型并不更密集。
The Propensity for Density in Feed-forward Models
- 在不同宽度的全连接、卷积和残差模型上测试剪枝
- 最大模型比最小模型宽50倍，但可剪枝比例基本一致
- 发现模型越大越能被剪枝，适合模型压缩研究者
训练神经网络完成任务时，是否会用到所有可用权重？我们通过在不同宽度的全连接、卷积和残差模型上进行剪枝实验来回答这一问题。结果表明，在不降低性能的前提下，可剪枝权重的比例几乎不受模型大小影响。增大模型宽度对剪枝后模型密度的影响极小，即使最大模型宽度是最小模型的50倍，其可剪枝比例仍保持稳定。我们探讨了三种可能解释该现象的假设。
原文摘要 · Abstract (English)
Does the process of training a neural network to solve a task tend to use all of the available weights even when the task could be solved with fewer weights? To address this question we study the effects of pruning fully connected, convolutional and residual models while varying their widths. We find that the proportion of weights that can be pruned without degrading performance is largely invariant to model size. Increasing the width of a model has little effect on the density of the pruned model relative to the increase in absolute size of the pruned network. In particular, we find substantial prunability across a large range of model sizes, where our biggest model is 50 times as wide as our smallest model. We explore three hypotheses that could explain these findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。