通过压缩张量并行通信，显著降低大模型推理延迟。
Communication Compression for Tensor Parallel LLM Inference
- 采用细粒度量化压缩跨加速器通信数据，压缩比达3.5-4.5倍
- 实现最高2倍的首次生成时间（TTFT）减少，性能损失极小
- 适合追求低延迟的大模型部署场景
大语言模型（LLMs）参数和运算量高达数百亿，为提升推理速度，常通过多种模型并行策略部署在多个硬件加速器上。本文聚焦张量并行策略，提出通过压缩跨加速器通信来降低延迟。利用细粒度量化技术，对部分激活值进行压缩，压缩比达3.5至4.5倍。所提方法可使首次生成时间（TTFT）最多减少2倍，且模型性能下降可忽略不计。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have pushed the frontier of artificial intelligence but are comprised of hundreds of billions of parameters and operations. For faster inference latency, LLMs are deployed on multiple hardware accelerators through various Model Parallelism strategies. Our paper looks into the details on one such strategy - Tensor Parallel - and proposes to reduce latency by compressing inter-accelerator communication. We leverage fine grained quantization techniques to compress selected activations by 3.5 - 4.5x. Our proposed method leads up to 2x reduction of time-to-first-token (TTFT) with negligible model performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。