将张量并行通信数据从16位压缩至4.2位，几乎不损失模型性能。
Towards Low-bit Communication for Tensor Parallel LLM Inference
- 利用通信特征中的稳定异常值，设计低比特量化方法。
- 在Gemma 2 27B和Llama 2 13B上分别保持98.0%和99.5%性能。
- 适合大规模LLM推理中需降低通信开销的场景。
张量并行虽能提升服务器大语言模型（LLM）推理效率，但随模型规模扩大，设备分布增多，通信开销显著上升。现有量化方法通常避免对张量并行所需通信的特征进行量化。本文利用通信特征中稳定的异常值，提出一种量化方法，使平均通信值从16位降至4.2位，同时几乎完全保留原始性能。例如，在所有评估任务上，该方法分别保持Gemma 2 27B和Llama 2 13B约98.0%和99.5%的原始性能。
原文摘要 · Abstract (English)
Tensor parallelism provides an effective way to increase server large language model (LLM) inference efficiency despite adding an additional communication cost. However, as server LLMs continue to scale in size, they will need to be distributed across more devices, magnifying the communication cost. One way to approach this problem is with quantization, but current methods for LLMs tend to avoid quantizing the features that tensor parallelism needs to communicate. Taking advantage of consistent outliers in communicated features, we introduce a quantization method that reduces communicated values on average from 16 bits to 4.2 bits while preserving nearly all of the original performance. For instance, our method maintains around 98.0% and 99.5% of Gemma 2 27B's and Llama 2 13B's original performance, respectively, averaged across all tasks we evaluated on.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。