通过跨层共享基础向量，大幅压缩大模型参数量。
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression
- 用共享基础向量+独有系数表示不同层权重矩阵。
- 在高压缩比下仍保持模型性能，优于现有SVD方法。
- 适合资源受限场景下的大模型部署与推理加速。
大语言模型（LLMs）取得了显著突破，但其庞大的参数量导致推理时需要大量内存存储，限制了在实际应用中的部署。为减少内存占用，奇异值分解（SVD）为近似权重矩阵、压缩LLMs提供了有效途径。本文进一步探索在不同层间通过SVD实现参数共享，以提升压缩效率。具体地，将不同层的权重矩阵分解并表示为一组共享的基础向量与各自独有系数的线性组合。研究了压缩时权重矩阵类型及层选择对性能的影响。全面实验表明，该方法在大压缩比下优于当前最先进的基于SVD的压缩方案和参数共享技术。代码已开源：https://github.com/TUDa-HWAI/Basis_Sharing。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable breakthroughs. However, the huge number of parameters in LLMs require significant amount of memory storage in inference, which prevents their practical deployment in many applications. To reduce memory storage of LLMs, singular value decomposition (SVD) provides a promising solution to approximate weight matrices for compressing LLMs. In this paper, we take a step further to explore parameter sharing across different layers with SVD to achieve more effective compression for LLMs. Specifically, weight matrices in different layers are decomposed and represented as a linear combination of a set of shared basis vectors and unique coefficients. The types of weight matrices and the layer selection for basis sharing are examined when compressing LLMs to maintain the performance. Comprehensive experiments demonstrate that Basis Sharing outperforms state-of-the-art SVD-based compression approaches and parameter sharing techniques, especially under large compression ratios. Code is available at: https://github.com/TUDa-HWAI/Basis_Sharing
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。