发现稀疏模型失效主因是激活方差下降，提出无需更新权重的新方法恢复性能。
Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations
- 通过抑制激活方差下降来缓解信号坍塌，而非优化权重重选
- 在保留20%参数下，将ResNeXt101在ImageNet准确率从4.1%提升至78.9%
- 适用于资源受限设备部署，尤其适合追求高精度稀疏模型的研究者
神经网络剪枝对降低模型复杂度、实现资源受限硬件部署至关重要。现有研究常将剪枝后性能下降归因于关键参数的移除，但我们发现根本原因是各层激活方差减小导致的信号坍塌。当前的一次性剪枝方法侧重于权重选择策略，依赖计算成本高的二阶近似。相反，我们证明缓解信号坍塌比优化权重选择更为关键。为此提出REFLOW方法，不更新可训练权重即可解决信号坍塌问题，在原始参数空间中挖掘出高质量稀疏子网络。该方法使基于幅值的剪枝达到顶尖性能：在仅保留20%权重的情况下，将ResNeXt101在ImageNet上的准确率从不足4.1%恢复至78.9%，超越现有最佳方法。
原文摘要 · Abstract (English)
Neural network pruning is essential for reducing model complexity to enable deployment on resource constrained hardware. While performance loss of pruned networks is often attributed to the removal of critical parameters, we identify signal collapse a reduction in activation variance across layers as the root cause. Existing one shot pruning methods focus on weight selection strategies and rely on computationally expensive second order approximations. In contrast, we demonstrate that mitigating signal collapse, rather than optimizing weight selection, is key to improving accuracy of pruned networks. We propose REFLOW that addresses signal collapse without updating trainable weights, revealing high quality sparse sub networks within the original parameter space. REFLOW enables magnitude pruning to achieve state of the art performance, restoring ResNeXt101 accuracy from under 4.1% to 78.9% on ImageNet with only 20% of the weights retained, surpassing state of the art approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。