用2的幂次量化压缩大模型，显著节省内存并提速。
Power-of-Two Quantization-Aware-Training (PoT-QAT) in Large Language Models (LLMs)
- 权重仅保留2的幂次，用位移替代乘法，降低计算开销。
- 训练后困惑度提升66%，与原模型差异仅1%的BERT分数。
- 内存减少87.5%，推理速度提升3-10倍，适合边缘部署。
近年来，大语言模型参数量呈指数级增长，从GPT-2的15亿到GPT-3的1750亿，甚至可能超过万亿。这给边缘设备部署带来严峻挑战，因其内存与算力极为有限。本文研究一种仅允许2的幂次（PoT）表示权重的特殊量化方法，仅需存储指数部分即可大幅节省内存，更重要的是，通过位移操作替代乘法，显著降低计算开销。为缓解严格量化带来的性能下降，引入量化感知训练（QAT）进行额外训练。在GPT-2 124M上的实验表明，经训练后的PoT模型困惑度相比未训练版本提升66%，且与原始GPT-2的BERT-Score差异仅为1%。该方法预计可实现87.5%的内存节省，并使推理速度提升3-10倍。
原文摘要 · Abstract (English)
In Large Language Models (LLMs), the number of parameters has grown exponentially in the past few years, e.g., from 1.5 billion parameters in GPT-2 to 175 billion in GPT-3 to possibly more than trillion in higher versions. This raises a significant challenge for implementation, especially for Edge devices. Unlike cloud computing, memory and processing power for Edge devices are very limited, which necessitates developing novel ideas to make such applications feasible. In this work, we investigate compressing weights with a special quantization that limits numbers to only power-of-two (PoT). This helps save a huge amount of memory as only exponents need to be stored, more importantly, it significantly reduces processing power by replacing costly multiplication with low cost bit shifting. To overcome performance loss due to this strict quantization, we investigate Quantization Aware Training (QAT) to enhance performance through additional training. Results on GPT-2 124M show a major enhancement for quantized PoT model after additional training, with a perplexity enhancement of 66% and BERT-Score loss to baseline GPT-2 of 1%. The memory saving is estimated to be 87.5% while the inference speed is expected to be 3-10x faster with PoT quantization versus full-precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。