通过正交变换提升大模型结构化矩阵压缩效果
ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations
- 利用权重矩阵的正交变换不变性优化压缩
- 可在不微调前提下显著提升结构化矩阵适配度
- 适用于多种高效投影结构，适合模型部署场景
大型语言模型在自然语言处理任务中表现卓越，但需大量计算与内存资源。结构化矩阵表示是减少模型参数量的有前景方法。然而，未经微调的预训练模型权重难以被结构化矩阵精确表示。为此，我们利用大模型输出对权重矩阵某些正交变换保持不变的特性，识别出能显著提升结构化类别中权重可压缩性的变换。该方法适用于支持高效投影操作的各类结构化矩阵，代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by structured matrices without any fine-tuning. To overcome this issue, we utilize the fact that LLM output is invariant under certain orthogonal transformations of weight matrices. This insight can be leveraged to identify transformations that significantly improve the compressibility of weights within structured classes. The proposed approach is applicable to various types of structured matrices that support efficient projection operations. Code is available at https://github.com/GrishKate/ProcrustesGPT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。