通过补偿机制实现零开销层剪枝，显著提升大模型压缩性能。
Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation
- 采用离线权重重缩放补偿剪枝带来的隐藏状态幅值差异。
- 在剪掉5层后，困惑度接近减半，问答性能保留93.19%。
- 无需训练即可集成，适合追求高效部署的模型压缩场景。
层剪枝已成为压缩大语言模型并实现与剪枝比例成正比加速的有前景技术。本文发现，移除任意一层会引发隐藏状态的显著幅值差距，导致性能大幅下降。为此，我们提出Prune&Comp，一种无需训练的即插即用层剪枝方案，通过幅值补偿来消除该差距。具体而言，先估计层移除引起的幅值偏差，再通过离线重缩放剩余权重进行补偿，不引入任何运行时开销。进一步结合迭代剪枝策略，当与剪枝-补偿循环集成时，Prune&Comp持续提升现有层剪枝指标。例如，在使用主流块影响度量剪掉LLaMA-3-8B的5层后，困惑度近乎减半，问答性能保留93.19%，优于基线4.01%。
原文摘要 · Abstract (English)
Layer pruning has emerged as a promising technique for compressing large language models (LLMs) while achieving acceleration proportional to the pruning ratio. In this work, we identify that removing any layer induces a significant magnitude gap in hidden states, resulting in substantial performance degradation. To address this issue, we propose Prune&Comp, a novel plug-and-play layer pruning scheme that leverages magnitude compensation to mitigate such gaps in a training-free manner. Specifically, we first estimate the magnitude gap caused by layer removal and then eliminate this gap by rescaling the remaining weights offline, with zero runtime overhead incurred. We further demonstrate the advantages of Prune&Comp through an iterative pruning strategy. When integrated with an iterative prune-and-compensate loop, Prune&Comp consistently enhances existing layer pruning metrics. For instance, when 5 layers of LLaMA-3-8B are pruned using the prevalent block influence metric, Prune&Comp nearly halves the perplexity and retains 93.19\% of the original model's question-answering performance, outperforming the baseline by 4.01%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。