通过全局稀疏分布优化,高效剪枝大模型同时保持高精度。
Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
- 构建超网并迭代搜索最优稀疏分布,实现端到端剪枝。
- 在保留97%原始性能前提下,剪掉Llama-3.1-70B一半参数。
- 适合需要极致压缩且不牺牲性能的模型部署场景。
结构化剪枝可提升大语言模型的硬件无关推理效率,但常难以维持相近性能。局部剪枝虽能逐层高效压缩,却忽略全局拓扑结构;而传统全局剪枝多采用两阶段范式,先评估子结构重要性再执行全局剪枝,忽视结构间依赖关系,无法实现端到端优化。为此,我们提出Týr-the-Pruner,一种基于搜索的端到端全局结构化剪枝框架。该框架通过在不同稀疏率下反复对每层进行局部剪枝,构建超网,核心目标是确定在目标总体稀疏率下的最优稀疏分布。具体而言,引入有效局部剪枝方法和期望误差累积策略以提升超网质量,并采用粗到细粒度的迭代剪枝-搜索策略,确保高效收敛。实验表明,Týr-the-Pruner达到当前最优结构化剪枝效果:在保留97%稠密模型性能的前提下,成功移除Llama-3.1-70B中50%的参数。代码将开源至https://github.com/AMD-AGI/Tyr-the-Pruner。
原文摘要 · Abstract (English)
Structural pruning enhances hardware-agnostic inference efficiency for large language models (LLMs) yet often fails to maintain comparable performance. Local pruning performs efficient layer-by-layer compression but ignores global topology. Although global pruning aims to identify an optimal sparse model, intuitive methods typically adopt a two-stage paradigm that first evaluates substructure saliency and then applies global pruning, which ignores inter-structure dependencies and fails to achieve end-to-end optimization. To address these limitations, we propose Týr-the-Pruner, an efficient end-to-end search-based global structural pruning framework. This framework constructs a supernet by repeatedly applying local pruning across a range of sparsity ratios to each layer in an LLM, with the core goal of determining the optimal sparsity distribution under a target overall sparsity ratio. Concretely, we introduce an effective local pruning and an expectation error accumulation approach to improve supernet construction. Furthermore, we employ an iterative prune-and-search strategy with coarse-to-fine sparsity granularity to ensure efficient search convergence. Experimental results show that Týr-the-Pruner achieves state-of-the-art structural pruning, retaining 97% of the dense model's performance while removing a challenging 50% of Llama-3.1-70B's parameters. Code will be available at https://github.com/AMD-AGI/Tyr-the-Pruner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。