用2的幂权重压缩大模型，提升边缘设备运行效率。
Power-of-Two (PoT) Weights in Large Language Models (LLMs)
- 用2的幂次作为权重值，将乘法转为位移运算，降低计算开销。
- 4到6比特下损失仅上升1.3到0.88，精度损失小。
- 适合资源受限设备部署，如边缘端语言模型推理。
神经网络复杂度随参数量激增而快速上升，大型语言模型（LLMs）参数量从GPT-2的15亿增长至GPT-3的1750亿，带来实现挑战，尤其在内存和算力受限的边缘设备上。本文研究通过一种特殊量化方式——2的幂（PoT）量化，对线性层权重和Transformer表进行压缩。该方法不仅减少存储占用，更重要的是通过将乘法替换为位移操作，显著降低计算开销。初步实验在基于Shakespeare数据集的Nano-GPT上验证，随后扩展至1.24亿参数的GPT-2模型。结果显示，在4至6比特表示幂级数时，交叉熵损失增加约[1.3–0.88]，表现令人鼓舞。
原文摘要 · Abstract (English)
Complexity of Neural Networks is increasing rapidly due to the massive increase in model parameters. Specifically, in Large Language Models (LLMs), the number of model parameters has grown exponentially in the past few years, for example, from 1.5 billion parameters in GPT2 to 175 billion in GPT3. This raises a significant challenge for implementation, especially for Edge devices where memory and processing power are very limited. In this work, we investigate reducing LLM complexity with special type of quantization, power of two (PoT), for linear layers weights and transformer tables. PoT not only provides memory reduction but more importantly provides significant computational reduction through converting multiplication to bit shifting. We obtained preliminary results of PoT quantization on Nano-GPT implementation using Shakespeare dataset. We then extended results to 124-M GPT-2 model. The PoT quantization results are shown to be very promising with cross entropy loss degradation $\approx$[1.3-0.88] with number of bits range [4-6] to represent power levels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。