提出双重要性保护SVD压缩方法，提升大模型压缩后性能
DipSVD: Dual-importance Protected SVD for Efficient LLM Compression
- 通过通道加权白化保护每层权重矩阵的关键奇异向量
- 让非关键层承担更多压缩负担，减少对核心层影响
- 在高压缩比下仍保持优异性能，适合部署资源受限场景
大型语言模型（LLMs）日益增长的计算需求和部署成本催生了多种压缩方法。相比量化和非结构化剪枝，基于奇异值分解（SVD）的压缩具有更好的硬件兼容性和理论保障。然而，现有SVD方法仅关注整体矩阵差异，忽视对矩阵内关键组件的保护，导致压缩后模型性能下降。本文提出一种双层级重要性保护机制：(1) 局部重要性保护——通过通道加权数据白化保留每个权重矩阵中最关键的奇异向量；(2) 全局重要性保护——采用启发式或优化方法使不重要的层承担更大压缩压力，从而最小化压缩对关键层的影响。大量实验表明，DipSVD在多个基准测试中优于现有SVD压缩方法，在高模型压缩比下表现尤为出色。
原文摘要 · Abstract (English)
The ever-increasing computational demands and deployment costs of large language models (LLMs) have spurred numerous compressing methods. Compared to quantization and unstructured pruning, SVD compression offers superior hardware compatibility and theoretical guarantees. However, existing SVD-based methods focus on the overall discrepancy between the original and compressed matrices while overlooking the protection of critical components within the matrix, which leads to inferior performance in the compressed models. This paper proposes a dual-level importance protection mechanism to enhance SVD-based compression methods: (1) local importance protection: preserving the most critical singular vectors within each weight matrix through channel-weighted data whitening; and (2) global importance protection: enabling less important layers to bear a greater portion of the compression burden through either a heuristic or optimization-based approach, thereby minimizing the impact of compression on critical layers. Extensive experiments demonstrate that DipSVD outperforms existing SVD-based compression approaches across multiple benchmarks, achieving superior model performance especially at high model compression ratios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。