无需训练即可剪枝大模型，用线性操作替代层块
ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
- 用线性变换替代被剪枝的Transformer块，仅需小规模校准数据
- 在不训练情况下实现25%剪枝率，保持90%原始性能
- 适合部署时快速压缩模型，无需额外参数或微调
我们提出ReplaceMe，一种通用的无训练深度剪枝方法，可将Transformer块替换为线性操作，在低压缩比下保持高性能。与传统需要再训练或微调的方法不同，该方法仅需少量校准数据来估计一个近似剪枝块的线性映射，该映射可无缝合并到剩余块中，无需增加额外参数。实验表明，ReplaceMe持续优于其他无训练剪枝方法，并在性能上媲美需大量重训练和结构修改的前沿方法。应用于多个大语言模型（LLMs），ReplaceMe在不进行任何训练或修复步骤的情况下，实现最高25%的剪枝率，同时在公开基准上保持约90%的原始性能，计算开销极低。我们开源了实现ReplaceMe及多种先进深度剪枝技术的库，地址为https://github.com/mts-ai/ReplaceMe。
原文摘要 · Abstract (English)
We introduce ReplaceMe, a generalized training-free depth pruning method that effectively replaces transformer blocks with a linear operation, while maintaining high performance for low compression ratios. In contrast to conventional pruning approaches that require additional training or fine-tuning, our approach requires only a small calibration dataset that is used to estimate a linear transformation, which approximates the pruned blocks. The estimated linear mapping can be seamlessly merged with the remaining transformer blocks, eliminating the need for any additional network parameters. Our experiments show that ReplaceMe consistently outperforms other training-free approaches and remains highly competitive with state-of-the-art pruning methods that involve extensive retraining/fine-tuning and architectural modifications. Applied to several large language models (LLMs), ReplaceMe achieves up to 25\% pruning while retaining approximately 90\% of the original model's performance on open benchmarks - without any training or healing steps, resulting in minimal computational overhead. We provide an open-source library implementing ReplaceMe alongside several state-of-the-art depth pruning techniques, available at https://github.com/mts-ai/ReplaceMe
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。