用进化算法自动剪枝大模型，速度快且省训练数据。
DarwinLM: Evolutionary Structured Pruning of Large Language Models
- 通过突变生成多个子模型，择优保留适应性强的结构。
- 在剪枝后仅用1/5数据就能超越现有方法，压缩后仍保持高性能。
- 适合追求高效部署的大模型应用，尤其资源受限场景。
大语言模型在自然语言任务中表现优异，但其庞大的计算开销限制了在实时场景中的广泛应用。结构化剪枝可通过压缩模型实现端到端加速，不受硬件限制。不同模型组件对剪枝的敏感性各异，需非均匀压缩。现有方法不仅需识别有效子结构，还需考虑剪枝后的再训练。为此，我们提出达尔文剪枝（DarwinLM），一种面向训练的结构化剪枝方法。该方法基于进化搜索，每代通过突变生成多个子模型，并根据适应度选择最优个体。为评估再训练效果，我们在后代群体中引入轻量级多步训练流程，逐步增加训练词元数并淘汰表现差的模型。在Llama-2-7B、Llama-3.1-8B和Qwen-2.5-14B-Instruct上验证，达尔文剪枝达到当前结构化剪枝最优性能。例如，在再训练阶段仅需现有方法1/5的数据量，即超越ShearedLlama。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved significant success across various NLP tasks. However, their massive computational costs limit their widespread use, particularly in real-time applications. Structured pruning offers an effective solution by compressing models and directly providing end-to-end speed improvements, regardless of the hardware environment. Meanwhile, different components of the model exhibit varying sensitivities towards pruning, calling for non-uniform model compression. However, a pruning method should not only identify a capable substructure, but also account for post-compression training. To this end, we propose DarwinLM, a method for training-aware structured pruning. DarwinLM builds upon an evolutionary search process, generating multiple offspring models in each generation through mutation, and selecting the fittest for survival. To assess the effect of post-training, we incorporate a lightweight, multistep training process within the offspring population, progressively increasing the number of tokens and eliminating poorly performing models in each selection stage. We validate our method through extensive experiments on Llama-2-7B, Llama-3.1-8B and Qwen-2.5-14B-Instruct, achieving state-of-the-art performance for structured pruning. For instance, DarwinLM surpasses ShearedLlama while requiring 5x less training data during post-compression training. Code is at: https://github.com/IST-DASLab/DarwinLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。