通过多维渐进剪枝，平衡大模型结构，提升压缩效率与精度。
MultiPruner: Balanced Structure Removal in Foundation Models
- 从残差块、MLP通道、注意力头三维度逐步剪枝，保持结构平衡。
- 在不微调情况下实现更高压缩比，下游任务零样本精度更优。
- 适合需要高效部署大模型的工程场景，尤其关注推理性能优化者。
近期主流的大规模预训练模型（LPMs）剪枝方法表明，无需训练即可移除Transformer中非关键残差块是可行的，且效果优于以往无训练剪枝方法。受此启发，我们扩展了BlockPruner（Zhong et al., 2024），提出MultiPruner，一种采用多维、迭代、细粒度剪枝策略的新方法，超越了现有无训练剪枝技术。在MultiPruner中，多维剪枝通过依次压缩三个维度——残差块、多层感知机（MLP）通道、注意力头——恢复剪枝后模型的结构平衡性。该方法在不进行微调的情况下，相比其他技术显著提升了下游任务的零样本精度，并进一步提高了模型压缩率，生成计算和内存开销更少的压缩模型。大量实验验证了该方法在多种大规模预训练模型上的优势。代码与剪枝配置见：https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning。
原文摘要 · Abstract (English)
Recently, state-of-the-art approaches for pruning large pre-trained models (LPMs) have demonstrated that the training-free removal of non-critical residual blocks in Transformers is viable for reducing model size, achieving results that outperform previous training-free pruning approaches. Motivated by these findings, we extend BlockPruner (Zhong et al., 2024) and propose MultiPruner, a pruning approach that surpasses recent training-free pruning methods by adopting a multidimensional, iterative, fine-grained pruning strategy. In MultiPruner, multidimensional pruning reinstates the structural balance in block-pruned models by sequentially compressing along three dimensions: i) residual blocks, ii) channels of multilayer perceptrons (MLP), and iii) attention heads. This solution enhances zero-shot accuracy on downstream tasks compared to other techniques while improving model compression ratios, producing compressed models with fewer computing and memory requirements. Extensive experiments demonstrate the advantages of the proposed method across various large pre-trained models. The code and pruning configurations are available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。