通过熵编码压缩大模型权重，显著降低边缘设备存储与推理延迟。
EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices
- 混合量化+熵编码，提升权重可压缩性。
- 4比特下压缩率提升11.3倍，存储节省达65%。
- 无需重训练，适合部署在Jetson等边缘设备。
大型语言模型（LLMs）在各类任务中表现优异，但在边缘设备上面临存储与计算挑战。本文提出EntroLLM，一种结合混合量化与熵编码的压缩框架，在保持精度的同时减少存储占用。采用无符号与非对称量化，张量级量化带来熵减效应，使权重可压缩性显著提升：相比现有方法，8比特下霍夫曼编码压缩率提高7倍，4比特下提升11.3倍。霍夫曼编码进一步降低内存带宽需求，配合并行解码策略实现高效权重读取，延迟极低。在边缘规模LLM（smolLM-1.7B、phi3-mini-4k、mistral-7B）上的实验表明，相比uint8模型最多节省30%存储，相比uint4模型节省高达65%，在内存受限的NVIDIA JETSON P3450设备上推理速度提升31.9%-146.6%。EntroLLM无需重训练，兼容现有后训练量化流程，适用于实际边缘LLM部署。
原文摘要 · Abstract (English)
Large Language Models (LLMs) achieve strong performance across tasks, but face storage and compute challenges on edge devices. We propose EntroLLM, a compression framework combining mixed quantization and entropy coding to reduce storage while preserving accuracy. We use a combination of unsigned and asymmetric quantization. Tensor-level quantization produces an entropy-reducing effect, increasing weight compressibility, and improving downstream Huffman encoding by $7\times$ (8-bit) and $11.3\times$ (4-bit) over state-of-the-art methods. Huffman coding further reduces memory bandwidth demands, while a parallel decoding strategy enables efficient weight retrieval with minimal latency. Experiments on edge-scale LLMs (smolLM-1.7B, phi3-mini-4k, mistral-7B) show up to $30\%$ storage savings over uint8 and $65\%$ over uint4 models, with $31.9-146.6\%$ faster inference on memory-limited devices like the NVIDIA JETSON P3450. EntroLLM requires no retraining and is compatible with existing post-training quantization pipelines, making it practical for edge LLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。