提出新型量化框架ELUTQ，提升边缘设备上大模型的推理效率与精度。
ELUTQ: Optimizing Quantization Accuracy under LUT-Based Computation for Edge LLMs
- 采用分层线性量化（HLQ）格式，降低低比特下的量化误差。
- 2比特下运行LLaMA3.1-8B比AWQ快1.5倍，且无需重新训练。
- 仅需64GB CPU内存和48GB显存即可完成70B模型量化，适合边缘部署。
权重量化能有效降低内存占用，实现大语言模型在边缘设备上的部署，但现有硬件友好方法多依赖均匀量化，在低比特设置下存在权重分布拟合差、反量化开销高的问题。本文提出ELUTQ，一种高效量化框架，包含新型量化格式——分层线性量化（HLQ）。HLQ能更好捕捉权重统计特性，并通过位串行查找表（LUT）基矩阵乘法消除反量化开销。在低比特设置下显著提升模型精度，且性能接近基于量化感知训练（QAT）的方法，无需重新训练权重。此外，优化的量化流水线使LLaMA 3.1-70B可在仅64 GB CPU内存和48 GB VRAM条件下完成量化，大幅降低大规模模型量化对硬件的要求。为支持边缘设备高效部署，ELUTQ设计高性能内核实现端到端推理。2比特的LLaMA3.1-8B在RTX 3090上相比AWQ实现1.5倍加速。代码已开源。
原文摘要 · Abstract (English)
Weight quantization effectively reduces memory consumption and enable the deployment of Large Language Models on edge devices, yet existing hardware-friendly methods often rely on uniform quantization, which suffers from poor weight-distribution fitting and high dequantization overhead under low-bit settings. In this paper, we propose ELUTQ, an efficient quantization framework featuring a novel quantization format termed Hierarchical Linear Quantization (HLQ). HLQ is designed to better capture the statistical characteristics of weights and eliminate dequantization overhead using Bit-serial LUT-based GEMM operations. HLQ significantly improves model accuracy under low-bit settings and achieves performance comparable to QAT methods without any retraining of the weights. Moreover, an optimized quantization pipeline is integrated into ELUTQ, enabling it to complete the quantization of LLaMA 3.1-70B using only 64 GB of CPU memory and 48 GB of VRAM, reducing the hardware requirements for large-scale model quantization. To enable efficient deployment on edge devices, ELUTQ designs high-performance kernels to support end-to-end inference. Our 2-bit LLaMA3.1-8B achieves 1.5x speedup over AWQ on RTX 3090. Code is available at https://github.com/Nkniexin/ELUTQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。