通过拼接不同微调版本的层来压缩大模型，保持性能同时减少25%参数
GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching
- 从多个微调版本中选择或合并层，动态组合最优结构
- Llama2-13B压缩后保留97.3%原性能，参数减少约25%
- 适合需要轻量化部署且不牺牲性能的研究与工程场景
大型语言模型在语言理解与生成方面表现出色，但其庞大的模型规模带来了部署与推理上的挑战。虽然结构化剪枝可降低计算成本，但现有方法多聚焦于单一模型剪枝。本文提出一种新策略：通过融合不同微调版本的层，实现模型压缩，从而保留原始模型能力并整合各微调版本的优势。将最优剪裁建模为零阶优化问题，搜索空间支持三种操作：（1）移除层，（2）从候选模型中选择层，（3）合并层。实验表明，该方法在Llama2-13B上实现约25%参数减少的同时，性能保持在原始水平的97.3%，显著优于当前最优方法。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in deployment and inference. While structured pruning of model parameters offers a promising way to reduce computational costs at deployment time, current methods primarily focus on single model pruning. In this work, we develop a novel strategy to compress models by strategically combining or merging layers from finetuned model variants, which preserves the original model's abilities by aggregating capabilities accentuated in different finetunes. We pose the optimal tailoring of these LLMs as a zero-order optimization problem, adopting a search space that supports three different operations: (1) Layer removal, (2) Layer selection from different candidate models, and (3) Layer merging. Our experiments demonstrate that this approach leads to competitive model pruning, for example, for the Llama2-13B model families, our compressed models maintain approximately 97.3\% of the original performance while removing $\sim25\%$ of parameters, significantly outperforming previous state-of-the-art methods. The code is available at https://github.com/Guinan-Su/auto-merge-llm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。