提出自动连续层剪枝方法,高效压缩大模型且保持高精度。
The Structural Scalpel: Automated Contiguous Layer Pruning for Large Language Models
- 用可微凹门控算法自动识别需剪枝的连续层段。
- 20%剪枝率下LLaMA3-70B性能保留95.34%,优于基线4.29%~30.52%。
- 适合需要部署大模型的边缘设备,兼容量化进一步压缩。
尽管大语言模型在多个领域取得突破性进展,但其庞大的模型规模和高昂的计算成本给资源受限的边缘设备部署带来挑战。为此,层剪枝被提出以通过直接移除冗余层来降低计算开销。然而,现有方法通常依赖手工设计的度量标准评估并逐层移除,忽略了层间的依赖关系,可能破坏模型的信息流并严重损害性能。为此,我们提出CLP,一种新颖的连续层剪枝框架,包含两项关键创新:基于梯度优化的可微凹门控算法,可自动识别最优连续剪枝层段;以及仅微调剪枝段邻近层的截断端点调优策略,有效恢复模型性能。在多种模型架构(包括LLaMA2、LLaMA3和Qwen)和规模(从70亿到700亿参数)上的大量实验表明,CLP显著优于现有最先进基线。例如,在20%剪枝率下,CLP在LLaMA3-70B上实现平均95.34%的性能保留,优于基线4.29%至30.52%。此外,CLP可无缝与量化结合,实现进一步压缩,仅带来轻微性能损失。
原文摘要 · Abstract (English)
Although large language models (LLMs) have achieved revolutionary breakthroughs in many fields, their large model size and high computational cost pose significant challenges for practical deployment on resource-constrained edge devices. To this end, layer pruning has been proposed to reduce the computational overhead by directly removing redundant layers. However, existing layer pruning methods typically rely on hand-crafted metrics to evaluate and remove individual layers, while ignoring the dependencies between layers. This can disrupt the model's information flow and severely degrade performance. To address these issues, we propose CLP, a novel continuous layer pruning framework that introduces two key innovations: a differentiable concave gate algorithm that automatically identifies the best continuous layer segments for pruning via gradient-based optimization; and a cutoff endpoint tuning strategy that effectively restores model performance by fine-tuning only the layers adjacent to the pruned segments. Extensive experiments across multiple model architectures (including LLaMA2, LLaMA3 and Qwen) and sizes (from $7$B to $70$B parameters) show that CLP significantly outperforms existing state-of-the-art baselines. For example, at a pruning rate of $20\%$, CLP achieves an average performance retention of $95.34\%$ on LLaMA3-70B, outperforming baselines by $4.29\%$-$30.52\%$. Furthermore, CLP can be seamlessly combined with quantization to further compress the model with only a slight performance loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。