将大模型权重分解为稀疏、低秩和二值三部分，实现高效压缩且不损失性能。
SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models
- 将线性层权重拆分为稀疏、低秩和二值矩阵，三者协同压缩模型。
- 在Llama模型上达到50%压缩率时，困惑度降低36%，零样本任务准确率提升8.98%。
- 无需重训练，基于激活感知评分自动优化，适合部署场景的模型压缩。
大型语言模型（LLMs）的快速演进带来了巨大的计算与内存需求，部署困难。尽管网络剪枝等模型压缩技术有潜力缓解问题，但现有方法在高压缩比下往往难以保持良好性能。为此，我们提出SLaB框架，将每个线性层的权重分解为三个互补成分：稀疏矩阵、低秩矩阵和二值矩阵。该方法无需重训练，并利用激活感知剪枝分数指导分解过程。在Llama系列模型上的实验表明，SLaB达到当前最优性能，在50%压缩率下困惑度降低36%，零样本任务准确率相比基线提升8.98%。
原文摘要 · Abstract (English)
The rapid growth of large language models (LLMs) presents significant deployment challenges due to their massive computational and memory demands. While model compression, such as network pruning, offers potential solutions, most existing methods often fail to maintain good performance at high compression ratios. To address this, we propose SLaB, a novel framework that decomposes each linear layer weight into three complementary components: a sparse matrix, a low-rank matrix, and a binary matrix. SLaB eliminates the need for retraining and leverages activation-aware pruning scores to guide the decomposition process. Experiments on Llama-family models demonstrate that SLaB achieves state-of-the-art performance, reducing perplexity by up to 36% compared to existing methods at 50% compression and improving accuracy by up to 8.98% over the baseline on zero-shot tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。