不重训就能高效剪枝，保留深层特征的网络压缩新方法
Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework
- 用全局重构目标优化深层非线性特征，替代传统层内误差最小化
- 采用无海森矩阵的二阶优化，高效求解复杂优化问题
- 可兼容已有剪枝结果，适配残差网络与视觉变压器
我们提出SNOWS，一种无需重训练的一次性推理剪枝框架，旨在降低视觉网络的计算开销。现有主流方法通过最小化层内最小二乘重构误差进行剪枝，但未考虑网络深层表示。本文提出更全局的重构目标，引入深层非线性激活以更好逼近网络损失。该非线性目标带来更复杂的优化挑战，我们通过专用二阶优化框架高效求解。核心创新在于使用无海森矩阵优化，无需计算或存储完整海森矩阵即可获得精确牛顿下降步长。显著优势是可直接应用于任意先前方法生成的稀疏掩码，重新调整权重以利用深层特征非线性。在多个一次性剪枝基准上表现优异，涵盖残差网络及视觉变压器（ViT/B-16 和 ViT/L-16，分别含8600万和3.04亿参数）。
原文摘要 · Abstract (English)
We present SNOWS, a one-shot post-training pruning framework aimed at reducing the cost of vision network inference without retraining. Current leading one-shot pruning methods minimize layer-wise least squares reconstruction error which does not take into account deeper network representations. We propose to optimize a more global reconstruction objective. This objective accounts for nonlinear activations deep in the network to obtain a better proxy for the network loss. This nonlinear objective leads to a more challenging optimization problem -- we demonstrate it can be solved efficiently using a specialized second-order optimization framework. A key innovation of our framework is the use of Hessian-free optimization to compute exact Newton descent steps without needing to compute or store the full Hessian matrix. A distinct advantage of SNOWS is that it can be readily applied on top of any sparse mask derived from prior methods, readjusting their weights to exploit nonlinearities in deep feature representations. SNOWS obtains state-of-the-art results on various one-shot pruning benchmarks including residual networks and Vision Transformers (ViT/B-16 and ViT/L-16, 86m and 304m parameters respectively).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。