用可训练投影压缩Transformer，速度不降还能更小
Projected Compression: Trainable Projection for Efficient Transformer Compression
- 通过可训练投影模块降低模型权重维度
- 压缩后每词计算量与原模型一致，无额外开销
- 在高质量模型上优于传统剪枝重训方法
大语言模型规模持续增长以提升性能，但也带来推理时间与计算需求的增加。为此，我们提出一种新型模型压缩技术——投影压缩（Projected Compression），通过引入可训练投影模块减少模型参数量。首先训练额外的可学习投影权重，并保留对原始模型参数的访问；随后将这些投影合并为低维乘积矩阵，生成尺寸更小的标准Transformer模型。与需要额外计算开销的方法不同，本方法在每词计算步骤的浮点运算量（FLOPs）上与基础模型保持一致。实验表明，投影压缩在高质量模型上优于对比的硬性剪枝与重训练方法，且性能优势随令牌数量增加而扩大。
原文摘要 · Abstract (English)
Large language models have steadily increased in size to achieve improved performance; however, this growth has also led to greater inference time and computational demands. Consequently, there is rising interest in model size reduction methods. To address this issue, we propose Projected Compression, a novel model compression technique, that reduces model weights by utilizing projection modules. Specifically, we first train additional trainable projections weights and preserve access to all the original model parameters. Subsequently, these projections are merged into a lower-dimensional product matrix, resulting in a reduced-size standard Transformer-based model. Unlike alternative approaches that require additional computational overhead, our method matches the base model's per-token computation step in FLOPs. Experimental results show that Projected Compression outperforms the comparable hard pruning and retraining approach on higher quality models. Moreover, the performance margin scales well with the number of tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。