用优化理论实现大模型高效结构化剪枝，不降性能还省资源。
SPAP: Structured Pruning via Alternating Optimization and Penalty Methods
- 基于混合整数优化与惩罚法，自动决策剪枝节点。
- 30%稀疏度下推理提速1.29倍，内存减半且性能稳定。
- 适合需要部署轻量大模型的研究者和工程师。
大型语言模型（LLMs）的部署常受限于其巨大的计算和内存开销。结构化剪枝通过移除整个网络组件提供可行方案，但现有方法存在性能下降、依赖启发式指标或微调成本高等问题。为此，我们提出SPAP（Structured Pruning via Alternating Optimization and Penalty Methods），一种基于优化理论的新型高效结构化剪枝框架。SPAP将剪枝问题建模为混合整数优化问题，采用惩罚方法有效进行剪枝决策以最小化剪枝误差，并设计了一种针对可分解问题结构的交替最小化算法，实现高效的权重更新与性能恢复。在OPT、LLaMA-3/3.1/3.2和Qwen2.5模型上的大量实验表明，SPAP优于当前最先进方法，在30%稀疏度下实现1.29倍线性推理加速与成比例内存缩减，同时保持模型性能。本工作为大模型剪枝提供了实用且优化驱动的解决方案。
原文摘要 · Abstract (English)
The deployment of large language models (LLMs) is often constrained by their substantial computational and memory demands. While structured pruning presents a viable approach by eliminating entire network components, existing methods suffer from performance degradation, reliance on heuristic metrics, or expensive finetuning. To address these challenges, we propose SPAP (Structured Pruning via Alternating Optimization and Penalty Methods), a novel and efficient structured pruning framework for LLMs grounded in optimization theory. SPAP formulates the pruning problem through a mixed-integer optimization model, employs a penalty method that effectively makes pruning decisions to minimize pruning errors, and introduces an alternating minimization algorithm tailored to the splittable problem structure for efficient weight updates and performance recovery. Extensive experiments on OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models demonstrate SPAP's superiority over state-of-the-art methods, delivering linear inference speedups (1.29$\times$ at 30% sparsity) and proportional memory reductions. Our work offers a practical, optimization-driven solution for pruning LLMs while preserving model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。