通过输入嵌入的二阶信息实现无重训练的高效模型压缩
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
- 利用输入嵌入的二阶统计信息分解权重为稀疏与低秩矩阵之和
- 在不重训练情况下，压缩60%仍保持领先性能，加速比达1.37倍
- 适合追求高效率部署的大模型应用，尤其适合资源受限场景
大规模基础模型的兴起带来了深度学习的新时代，尽管实践成效显著，却也面临内存与计算成本过高的问题。为缓解此问题，研究者致力于无需重训练的后处理神经网络剪枝技术。尽管已有进展，现有方法在压缩率提升时性能仍持续下降。本文提出一种新方法OATS，利用输入嵌入的二阶矩信息，将模型权重分解为稀疏与低秩矩阵之和。该方法无需重训练，在Llama-3、Phi-3等大型语言模型及ViT、DINOv2等视觉变换器上，实现最高达60%的压缩率，同时相比同类剪枝模型获得高达1.37倍的CPU加速，达到当前最佳性能。
原文摘要 · Abstract (English)
The recent paradigm shift to large-scale foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by prohibitively expensive costs in terms of high memory consumption and compute. To mitigate these issues, there has been a concerted effort in post-hoc neural network pruning techniques that do not require costly retraining. Despite the considerable progress being made, existing methods often exhibit a steady drop in model performance as the compression increases. In this paper, we present a novel approach to compressing large transformers, coined OATS, that utilizes the second moment information in the input embeddings to decompose the model weights into a sum of sparse and low-rank matrices. Without any retraining, OATS achieves state-of-the-art performance when compressing models by up to $60\%$ on large language models such as Llama-3 and Phi-3 and vision transformers such as ViT and DINOv2 while delivering up to $1.37\times$ the CPU acceleration versus a model that was comparably pruned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。