简单剪掉最后25%层+微调头部和最后三层，性能超越多个同规模模型。
Reassessing Layer Pruning in LLMs: New Insights and Methods
- 剪掉模型最后25%层，仅微调lm_head和剩余最后三层。
- 剪枝后模型性能优于ChatGLM2-6B、Vicuna-7B等同类模型。
- 实测验证简单策略有效，适合资源受限场景部署。
尽管大语言模型在多个领域取得显著成果，其庞大规模带来巨大计算开销，限制了在资源受限环境中的部署。层剪枝作为一种直接移除模型层的压缩方法，能有效降低计算负担。然而，如何最优地进行层剪枝？复杂的选择指标是否真正有效？广泛使用的LoRA微调方法在剪枝后是否仍表现理想？为此，我们投入数千小时GPU算力，在多个维度上系统评测了大语言模型的层剪枝策略。结果表明，仅剪掉最后25%层，并对lm_head及剩余最后三层进行微调，即可获得优异性能。基于此方法，我们对Llama-3.1-8B-It进行剪枝，所得模型在多项任务中表现优于多个同规模主流模型，如ChatGLM2-6B、Vicuna-7B-v1.5、Qwen1.5-7B和Baichuan2-7B。相关模型权重已发布于Huggingface，代码开源至GitHub。
原文摘要 · Abstract (English)
Although large language models (LLMs) have achieved remarkable success across various domains, their considerable scale necessitates substantial computational resources, posing significant challenges for deployment in resource-constrained environments. Layer pruning, as a simple yet effective compression method, removes layers of a model directly, reducing computational overhead. However, what are the best practices for layer pruning in LLMs? Are sophisticated layer selection metrics truly effective? Does the LoRA (Low-Rank Approximation) family, widely regarded as a leading method for pruned model fine-tuning, truly meet expectations when applied to post-pruning fine-tuning? To answer these questions, we dedicate thousands of GPU hours to benchmarking layer pruning in LLMs and gaining insights across multiple dimensions. Our results demonstrate that a simple approach, i.e., pruning the final 25\% of layers followed by fine-tuning the \texttt{lm\_head} and the remaining last three layer, yields remarkably strong performance. Following this guide, we prune Llama-3.1-8B-It and obtain a model that outperforms many popular LLMs of similar size, such as ChatGLM2-6B, Vicuna-7B-v1.5, Qwen1.5-7B and Baichuan2-7B. We release the optimal model weights on Huggingface, and the code is available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。