一锤定音的模型压缩框架,兼顾精度与效率。
MOONSHOT : A Framework for Multi-Objective Pruning of Vision and Large Language Models

- 融合重建误差与二阶近似损失,实现多目标剪枝
- 在2:4稀疏下降低32.6%困惑度,提升4.9分准确率
- 适配大模型与视觉模型,无需重训练
权重剪枝是压缩大型神经网络的常用方法。本文聚焦于难以处理的训练后一次性剪枝场景,即在不进行任何微调的情况下压缩预训练模型。现有的一次性剪枝方法通常只优化单一目标,如层级重建误差或训练损失的二阶泰勒近似。我们发现,任一目标在不同架构和稀疏度下并非始终最优。为此,提出MOONSHOT框架,通过联合优化层级重建误差与二阶泰勒近似损失,将任意单目标剪枝方法扩展为多目标形式。该框架作为现有剪枝算法的封装器,提出高效计算逆海森矩阵的方法,在保持百亿参数模型可扩展性的同时,实现与顶尖剪枝器相当的效率。在Llama-3.2和Llama-2上,结合先进剪枝方法后,可在2:4稀疏下使C4困惑度降低32.6%,零样本分类平均准确率提升最高4.9分;在Vision Transformers上,70%稀疏下ImageNet-1k准确率提升超5点;在ResNet-50上,90%稀疏下获得4点增益。
原文摘要 · Abstract (English)
Weight pruning is a common technique for compressing large neural networks. We focus on the challenging post-training one-shot setting, where a pre-trained model is compressed without any retraining. Existing one-shot pruning methods typically optimize a single objective, such as a layer-wise reconstruction loss or a second-order Taylor approximation of the training loss. We highlight that neither objective alone is consistently the most effective across architectures and sparsity levels. Motivated by this insight, we propose MOONSHOT, a general and flexible framework that extends any single-objective pruning method into a multi-objective formulation by jointly optimizing both the layer-wise reconstruction error and second-order Taylor approximation of the training loss. MOONSHOT acts as a wrapper around existing pruning algorithms. To enable this integration while maintaining scalability to billion-parameter models, we propose modeling decisions and introduce an efficient procedure for computing the inverse Hessian, preserving the efficiency of state-of-the-art one-shot pruners. When combined with state-of-the-art pruning methods on Llama-3.2 and Llama-2 models, MOONSHOT reduces C4 perplexity by up to 32.6% at 2:4 sparsity and improves zero-shot mean accuracy across seven classification benchmarks by up to 4.9 points. On Vision Transformers, it improves accuracy on ImageNet-1k by over 5 points at 70% sparsity, and on ResNet-50, it yields a 4-point gain at 90% sparsity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。