提出序列级计算通信重叠技术，提升大模型推理效率
ISO: Overlap of Computation and Communication within Seqenence For LLM Inference
- 在序列层面实现计算与通信重叠，突破传统方法限制
- 30B/70B模型下预填充阶段提速35%（4090）和15%（A800）
- 适用于多请求场景，减少对微批次的依赖
在大语言模型推理中，Transformer结构与多GPU张量并行导致计算与通信串行执行，造成通信阶段计算资源严重闲置。现有优化方法主要通过重叠矩阵计算与通信、交错微批次来提升效率，但或无法实现理想重叠，或应用受限。本文提出一种新的序列级计算-通信重叠策略，不仅增强重叠程度，还降低使用约束。基于30B/70B模型的实验表明，该方法在4090显卡上预填充阶段可减少约35%耗时，在A800上减少约15%。
原文摘要 · Abstract (English)
In the realm of Large Language Model (LLM) inference, the inherent structure of transformer models coupled with the multi-GPU tensor parallelism strategy leads to a sequential execution of computation and communication. This results in substantial underutilization of computing resources during the communication phase. To mitigate this inefficiency, various techniques have been developed to optimize the use of computational power throughout the communication process. These strategies primarily involve overlapping matrix computations and communications, as well as interleaving micro-batches across different requests. Nonetheless, these approaches either fall short of achieving ideal overlap or impose certain limitations on their application. To overcome these challenges, this paper introduces a novel strategy for computation-communication overlap that operates at the sequence level. This method not only enhances the degree of overlap but also minimizes the constraints on its applicability. Experimental evaluations conducted using 30b/70b models have demonstrated significant improvements in efficiency. Specifically, the proposed technique has been shown to reduce time consumption by approximately 35% on 4090 GPU and by roughly 15% on A800 GPU during the prefill stage of LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。