PGB通过分组与置换实现BERT一次剪枝，高效压缩模型且保持精度。
PGB: One-Shot Pruning for BERT via Weight Grouping and Permutation
- 按权重分组并置换，一次性剪除注意力与前馈层中非重要结构
- 在BERT_BASE上实现高稀疏率，计算开销更低且精度优于现有方法
- 适合需要快速部署轻量级BERT的场景，尤其关注推理效率
大型预训练语言模型如BERT因体积庞大导致推理慢、内存占用高。现有压缩方法多依赖迭代剪枝与知识蒸馏，过程复杂且计算成本高。本文提出一种新型半结构化单次剪枝方法PGB（Permutation and Grouping for BERT），在保持精度的同时实现高压缩效率与稀疏度。PGB通过置换识别关键权重组，并在多头注意力与前馈层中整体剪除其余权重；若某层未形成重要组，则整层剔除以获得更紧凑模型。在BERT_BASE上的实验表明，PGB在计算成本和精度保留方面优于当前最优结构化剪枝方法。
原文摘要 · Abstract (English)
Large pretrained language models such as BERT suffer from slow inference and high memory usage, due to their huge size. Recent approaches to compressing BERT rely on iterative pruning and knowledge distillation, which, however, are often too complicated and computationally intensive. This paper proposes a novel semi-structured one-shot pruning method for BERT, called $\textit{Permutation and Grouping for BERT}$ (PGB), which achieves high compression efficiency and sparsity while preserving accuracy. To this end, PGB identifies important groups of individual weights by permutation and prunes all other weights as a structure in both multi-head attention and feed-forward layers. Furthermore, if no important group is formed in a particular layer, PGB drops the entire layer to produce an even more compact model. Our experimental results on BERT$_{\text{BASE}}$ demonstrate that PGB outperforms the state-of-the-art structured pruning methods in terms of computational cost and accuracy preservation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。