用低比特压缩减少大模型推理通信瓶颈，提速超3倍
Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
- 提出低比特压缩技术，优化张量并行通信
- 节点内通信速度提升3倍以上，首令牌耗时减半
- 适合追求高速推理的大模型部署场景
大语言模型规模持续增长，需借助多维并行分布式推理，但设备间通信开销显著，尤其在带宽受限情况下。本文提出Flash Communication，一种新型低比特压缩技术，有效缓解张量并行推理中的通信瓶颈。实验表明，该方法使节点内通信速度提升超过3倍，首令牌生成时间减少2倍，且几乎不影响模型精度。在多种最新大模型上验证了其有效性。
原文摘要 · Abstract (English)
The ever-increasing sizes of large language models necessitate distributed solutions for fast inference that exploit multi-dimensional parallelism, where computational loads are split across various accelerators such as GPU clusters. However, this approach often introduces significant communication overhead, especially on devices with limited bandwidth. In this paper, we introduce Flash Communication, a novel low-bit compression technique designed to alleviate the tensor-parallelism communication bottleneck during inference. Our method substantially boosts intra-node communication speed by more than 3x and reduces the time-to-first-token by 2x, with nearly no sacrifice in model accuracy. Extensive experiments on various up-to-date LLMs demonstrate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。