无需微调即可高效压缩大模型，保持高精度
Pruning Foundation Models for High Accuracy without Retraining
- 提出层级化一次性剪枝方法，同时优化多层权重
- 在不重训练情况下实现优于现有方法的精度表现
- 适用于Transformer与Mamba两类主流大模型
由于参数量庞大，基础模型或大语言模型（LLMs）的部署面临挑战。虽然剪枝是减少模型规模和加速推理的有效手段，但传统剪枝需在全量数据上进行多次迭代微调，消耗大量计算资源。为此，已有研究提出无重训练的一次性剪枝方法，但因缺乏数据驱动的微调，剪枝后性能常有下降。本文首次将层间剪枝问题形式化为联合优化目标，提出一个最优解，并设计了适用于无结构与半结构稀疏的后训练剪枝算法。在多种主流大模型（包括基于Transformer与Mamba架构）上的实验表明，该方法在不重训练条件下，性能显著优于当前最先进基线。代码已开源：https://github.com/piuzha/APT
原文摘要 · Abstract (English)
Despite the superior performance, it is challenging to deploy foundation models or large language models (LLMs) due to their massive parameters and computations. While pruning is a promising technique to reduce model size and accelerate the inference, the traditional pruning techniques can hardly be applied for LLMs as they need to finetune the model on the full dataset with multiple epochs consuming massive data and hardware resources. To deal with this problem, post-training pruning methods are proposed to prune LLMs in one-shot without retraining. However, their accuracy after pruning may suffer from certain performance degradation due to the lack of retraining with massive data. To address this issue, in this paper, we first formulate the post-training problem for layer-wise LLM compression to simultaneously prune multiple weights in LLMs. Next, we provide an optimal solution for this problem and design our post-training pruning algorithm for both unstructured and semi-structured sparsity. Our extensive experiments demonstrate the superior performance of the proposed methods in comparison to SOTA baselines across various LLM families including transformer-based LLMs and Mamba-based LLMs. Code link: https://github.com/piuzha/APT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。