剪掉大模型最后几层，能大幅降低内存和推理时间。
Large Language Models Are Overparameterized Text Encoders
- 剪掉模型最后部分层，仅用1000步微调即可实现轻量化。
- 最多可剪掉80%层，性能下降不超过5.1点，平均剪21%层仅降0.3点。
- 无需复杂代码，适合资源受限场景或快速部署文本编码任务。
大语言模型（LLM）在经过监督对比训练后，表现出强大的文本嵌入能力。然而其庞大的规模导致推理时间与内存占用过高。本文发现，在仅进行1000步监督训练前，剪掉模型最后p%的层，可成比例降低内存与推理开销。我们在四个先进LLM上评估文本嵌入任务，结果显示：最多可剪掉30%层而性能几乎不变，剪80%层时性能仅轻微下降。通过仅三行代码即可轻松集成至任意流水线。我们提出新剪枝策略L³Prune，基于初始损失选择最优剪枝配置：大模型变体平均剪21%参数，性能仅降0.3；小模型变体剪74%参数，性能降5.1。结果表明LLM在文本嵌入任务中存在过参数化，可被高效剪枝。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate strong performance as text embedding models when finetuned with supervised contrastive training. However, their large size balloons inference time and memory requirements. In this paper, we show that by pruning the last $p\%$ layers of an LLM before supervised training for only 1000 steps, we can achieve a proportional reduction in memory and inference time. We evaluate four different state-of-the-art LLMs on text embedding tasks and find that our method can prune up to 30\% of layers with negligible impact on performance and up to 80\% with only a modest drop. With only three lines of code, our method is easily implemented in any pipeline for transforming LLMs to text encoders. We also propose $\text{L}^3 \text{Prune}$, a novel layer-pruning strategy based on the model's initial loss that provides two optimal pruning configurations: a large variant with negligible performance loss and a small variant for resource-constrained settings. On average, the large variant prunes 21\% of the parameters with a $-0.3$ performance drop, and the small variant only suffers from a $-5.1$ decrease while pruning 74\% of the model. We consider these results strong evidence that LLMs are overparameterized for text embedding tasks, and can be easily pruned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。