综述大模型二值化量化技术，解决高资源消耗问题
Binary Neural Networks for Large Language Model: A Survey
- 从训练初期就用二值权重，降低模型存储与计算开销
- 对比传统量化方法，二值化可减少参数量至1/32
- 适合资源受限场景下的大模型部署与推理加速
大语言模型（LLMs）在自然语言处理领域广泛应用，如GPT-4和Llama。然而，随着模型参数规模呈指数增长，其带来显著的资源开销。低比特量化作为关键技术，通过降低模型参数、激活值和梯度的位宽，减少内存占用与计算需求。以往针对LLMs的量化方法主要采用事后量化（PTQ）和量化感知训练（QAT）。PTQ无需重新训练原模型，而QAT则在训练中优化精度以获得最优量化参数。BitNet团队提出一种全新方法：从模型训练初期即进行量化，使用低精度二值权重参与训练。这一思路催生了众多适用于大语言模型的二值量化技术。本文系统综述这些二值量化方法，涵盖深度神经网络中的相关技术，并深入探讨其在大语言模型中的应用、实现方式与贡献。
原文摘要 · Abstract (English)
Large language models (LLMs) have wide applications in the field of natural language processing(NLP), such as GPT-4 and Llama. However, with the exponential growth of model parameter sizes, LLMs bring significant resource overheads. Low-bit quantization, as a key technique, reduces memory usage and computational demands by decreasing the bit-width of model parameters, activations, and gradients. Previous quantization methods for LLMs have largely employed Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ does not require any retraining of the original model, while QAT involves optimizing precision during training to achieve the best quantization parameters. The BitNet team proposed a radically different approach, where quantization is performed from the start of model training, utilizing low-precision binary weights during the training process. This approach has led to the emergence of many binary quantization techniques for large language models. This paper provides a comprehensive review of these binary quantization techniques. Specifically, we will introduce binary quantization techniques in deep neural networks and further explore their application to LLMs, reviewing their various contributions, implementations, and applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。