融合稀疏与低秩压缩，让大模型更小更快无损
1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models
- 将稀疏剪枝和低秩分解统一为优化问题，迭代求解
- 70亿到700亿参数模型压缩50%仍零性能损失
- 无需重训练，适合部署在资源受限设备上
大语言模型在语言理解与生成方面表现卓越，但其广泛应用受限于巨大的带宽与计算需求。尽管剪枝和低秩逼近各自已展现出良好效果，但二者协同对大模型的潜力尚未充分探索。本文提出一种协同稀疏与低秩压缩方法（SSLC），结合两者优势：低秩逼近保留模型核心结构，稀疏优化剔除冗余权重，从而维持泛化能力。基于理论分析，我们将二者统一为联合优化问题，并通过迭代算法求解。在LLaMA与Qwen2.5（7B-70B）模型上的实验表明，SSLC无需额外训练即可持续超越单一方法，达到当前最佳性能。特别地，对Qwen2.5实现50%压缩且性能无下降，推理速度提升至少1.63倍，为高效部署提供切实可行方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable proficiency in language comprehension and generation; however, their widespread adoption is constrained by substantial bandwidth and computational demands. While pruning and low-rank approximation have each demonstrated promising performance individually, their synergy for LLMs remains underexplored. We introduce \underline{S}ynergistic \underline{S}parse and \underline{L}ow-Rank \underline{C}ompression (SSLC) methods for LLMs, which leverages the strengths of both techniques: low-rank approximation compresses the model by retaining its essential structure with minimal information loss, whereas sparse optimization eliminates non-essential weights, preserving those crucial for generalization. Based on theoretical analysis, we first formulate the low-rank approximation and sparse optimization as a unified problem and solve it by iterative optimization algorithm. Experiments on LLaMA and Qwen2.5 models (7B-70B) show that SSLC, without any additional training steps, consistently surpasses standalone methods, achieving state-of-the-arts results. Notably, SSLC compresses Qwen2.5 by 50\% with no performance drop and achieves at least 1.63$\times$ speedup, offering a practical solution for efficient LLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。