提出一种保持层间结构统一的大模型剪枝方法，提升推理效率和训练兼容性。
MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
- 通过极小化极大优化学习掩码，实现层间结构统一的剪枝
- 在保持高性能的同时，使剪枝后模型结构一致，优于现有最先进方法
- 适合需要高效部署和持续训练的大型语言模型应用
大型语言模型在各类自然语言任务中表现卓越，但其规模不断增长带来了部署与推理的挑战。结构化剪枝作为一种有效的模型压缩技术，因其能提升推理效率而受到关注。然而，多数基于优化的结构化剪枝方法为追求性能而牺牲了层间的结构统一性，导致结构异质，难以利用现成的推理加速工具，也阻碍了持续训练的高效配置。为此，我们提出一种基于极小化极大优化的掩码学习范式，在稀疏性正则化下优化掩码，以获得均匀的剪枝结构。大量实验表明，该方法可在保持高精度的同时确保剪枝后模型结构的统一性，显著优于现有最先进方法。
原文摘要 · Abstract (English)
The remarkable performance of large language models (LLMs) in various language tasks has attracted considerable attention. However, the ever-increasing size of these models presents growing challenges for deployment and inference. Structured pruning, an effective model compression technique, is gaining increasing attention due to its ability to enhance inference efficiency. Nevertheless, most previous optimization-based structured pruning methods sacrifice the uniform structure across layers for greater flexibility to maintain performance. The heterogeneous structure hinders the effective utilization of off-the-shelf inference acceleration techniques and impedes efficient configuration for continued training. To address this issue, we propose a novel masking learning paradigm based on minimax optimization to obtain the uniform pruned structure by optimizing the masks under sparsity regularization. Extensive experimental results demonstrate that our method can maintain high performance while ensuring the uniformity of the pruned model structure, thereby outperforming existing SOTA methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。