通过激活维度压缩实现大模型高效压缩，推理时无需精度损失。
ESPACE: Dimensionality Reduction of Activations for Model Compression
- 将激活投影到预校准主成分上，实现无损重训练。
- 对GPT3等模型压缩50%后，困惑度仅增0.18（GPT3-22B）。
- 适用于追求低延迟与高效率的部署场景，适合硬件优化。
我们提出ESPACE，一种基于激活维度压缩的大语言模型压缩技术。与以往聚焦权重的张量分解不同，ESPACE将激活投影到预校准的主成分集合上。该激活中心方法允许在不损失表达能力的情况下重训练大模型；推理时，权重分解可作为矩阵乘法结合律的副产品自然获得。理论部分提供了构造具有最优计算精度投影矩阵的方法。实验表明，ESPACE在保持小幅度精度下降的前提下，实现了对GPT3、Llama2和Nemotron4模型50%的压缩，其中GPT3-22B的困惑度仅增加0.18。在20%至40%的压缩率下，GPT3模型甚至超越基线，困惑度最低下降0.38（GPT3-8B）。此外，ESPACE还减少了GEMM执行时间和预填充推理延迟。与基于矩阵分解压缩Llama2-7B的现有方法对比，ESPACE标志着大模型张量分解压缩技术的阶段性突破。
原文摘要 · Abstract (English)
We propose ESPACE, an LLM compression technique based on dimensionality reduction of activations. Unlike prior works on weight-centric tensor decomposition, ESPACE projects activations onto a pre-calibrated set of principal components. The activation-centrality of the approach enables retraining LLMs with no loss of expressivity; while at inference, weight decomposition is obtained as a byproduct of matrix multiplication associativity. Theoretical results on the construction of projection matrices with optimal computational accuracy are provided. Experimentally, we find ESPACE enables 50% compression of GPT3, Llama2, and Nemotron4 models with small accuracy degradation, as low as a 0.18 perplexity increase on GPT3-22B. At lower compression rates of 20% to 40%, ESPACE drives GPT3 models to outperforming their baseline, by up to a 0.38 decrease in perplexity for GPT3-8B. ESPACE also reduces GEMM execution time and prefill inference latency on existing hardware. Comparison with related works on compressing Llama2-7B via matrix factorization shows that ESPACE is a first step in advancing the state-of-the-art in tensor decomposition compression of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。