arXiv:2510.03246cs.LGcs.AI2025-10

提出新型结构化剪枝方法,用更少显存实现大模型高效部署。

StructPrune: Structured Global Pruning asymptotics with $\mathcal{O}(\sqrt{N})$ GPU Memory

  • 分治策略将全局剪枝拆解为可并行的局部子问题,降低内存占用。
  • 理论推导出层间稀疏度分配规则,实测在百亿参数模型上保持性能。
  • 适合需要高硬件效率的大模型压缩场景,尤其适合资源受限部署。

剪枝对扩展大型语言模型至关重要。全局剪枝虽性能优异,但需$\\(mathcal{O}(N)\\)$显存,对十亿参数模型不可行;局部剪枝将显存降至单层水平,但忽略层间依赖,高稀疏度下表现不佳。结构化剪枝生成符合GPU优化的规整稀疏模式,但通常依赖全局剪枝,因局部优化易导致性能严重下降。为此,我们提出分治策略,将全局剪枝分解为模块间的协同子问题,每个子问题可适配有限显存。基于此,设计 extbf{STRUPRUNE}——一种基于ADMM的框架,将结构化稀疏嵌入剪枝过程,兼顾局部剪枝的内存效率与结构化方法的硬件兼容性。推导出结构化剪枝掩码的闭式解析解,给出层间稀疏度分配的明确规则,并构建能量驱动的渐近框架,得到软最大化形式的分配方案,简化优化同时适应不同层的重要性差异。实验表明,STRUPRUNE在保持全局结构化剪枝困惑度的同时,将显存需求从$\\mathcal{O}(N)\\)$降至$\\mathcal{O}(\sqrt{N})\\)$,实现百亿参数模型的实际部署。

原文摘要 · Abstract (English)

Pruning is critical for scaling large language models (LLMs). Global pruning achieves strong performance but requires $\mathcal{O}(N)$ memory, which is infeasible for billion-parameter models. Local pruning reduces GPU memory usage to that of a single layer by pruning layers independently, but it neglects inter-layer dependencies and often leads to suboptimal performance in high-sparsity regimes. Unlike unstructured pruning, structured pruning produces regular sparsity patterns that align well with GPU kernels and library optimizations, making it more hardware-efficient. However, structured pruning typically relies on global pruning, since structured patterns are more prone to severe performance degradation under local optimization. To jointly achieve structured pruning and the memory efficiency of local pruning, we propose a divide-and-conquer strategy that decomposes the global pruning problem into coordinated subproblems across different modules, each of which fits within limited GPU memory. Building on this idea, we design \textbf{STRUPRUNE}, an ADMM-based framework that integrates structured sparsity into the pruning process, combining the memory efficiency of local pruning with the hardware compatibility of structured methods. We derive a closed-form analytical solution for structured pruning masks that provides an explicit rule for layer-wise sparsity allocation, and further develop an energy-based asymptotic framework yielding a softmax-form allocation scheme that simplifies optimization while adapting to heterogeneous layer importance. Experiments demonstrate that STRUPRUNE matches the perplexity of global structured pruning while reducing memory cost from $\mathcal{O}(N)$ to $\mathcal{O}(\sqrt{N})$, enabling practical deployment at the billion-parameter scale.

模型剪枝结构化稀疏显存优化大模型部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。