通过相似通道共享权重，大幅压缩大模型参数量。
SWSC: Shared Weight for Similar Channel in LLM
- 按通道聚类相似权重，用代表向量替代整组
- 压缩后仍保持低精度下的模型性能
- 适合部署资源受限的大模型应用
大型语言模型（LLMs）推动了多个行业的发展，但其参数量持续增长带来了巨大的存储与计算负担，亟需探索模型压缩技术以减少参数并便于部署。我们提出SWSC方法，基于相似通道共享权重的理念，采用K-Means聚类算法对模型权重进行逐通道聚类，生成内部向量高度相似的簇。每个簇中选取一个代表性向量，近似替换簇内所有向量，显著降低模型权重参数数量。然而，近似恢复不可避免地导致性能下降。为此，我们在压缩前后对权重误差进行奇异值分解，保留较大的奇异值及其对应奇异向量，以补偿精度损失。实验结果表明，该方法在低精度条件下仍能有效保证压缩后LLM的性能。
原文摘要 · Abstract (English)
Large language models (LLMs) have spurred development in multiple industries. However, the growing number of their parameters brings substantial storage and computing burdens, making it essential to explore model compression techniques for parameter reduction and easier deployment. We propose SWSC, an LLM compression method based on the concept of Shared Weight for Similar Channel. It uses the K-Means clustering algorithm to cluster model weights channel-by-channel, generating clusters with highly similar vectors within each. A representative vector from each cluster is selected to approximately replace all vectors in the cluster, significantly reducing the number of model weight parameters. However, approximate restoration will inevitably cause damage to the performance of the model. To tackle this issue, we perform singular value decomposition on the weight error values before and after compression and retain the larger singular values and their corresponding singular vectors to compensate for the accuracy. The experimental results show that our method can effectively ensure the performance of the compressed LLM even under low-precision conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。