通过智能丢弃同步点，显著降低大模型分布式推理通信开销。
SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models
- 动态选择性跳过注意力层的同步操作，减少通信等待时间。
- 在8张GPU上运行LLaMA2-70B时，延迟降低约20%，精度损失小于1%。
- 适合追求高吞吐、低延迟的大模型部署场景，尤其适用于多卡集群。
随着大语言模型（LLMs）规模的迅速扩大，实现跨多个计算单元的高效分布式推理变得愈发关键。然而，主流分布式推理技术如张量并行带来的通信开销，严重制约了可扩展性和低延迟。为此，我们提出一种新优化技术——同步点丢弃（Sync-Point Drop, SPD），通过有选择地跳过注意力输出的同步操作，降低张量并行中的通信开销。具体而言，我们设计了一种块结构，使执行可在不依赖通信的情况下继续进行；同时，根据注意力块对模型精度的敏感程度，采用差异化的SPD策略。该方法有效缓解了通信瓶颈，同时最小化了推理过程中的精度下降，在多种分布式环境下提供了可扩展的解决方案：在8张GPU上对LLaMA2-70B进行推理时，总体延迟降低约20%，精度损失低于1%。
原文摘要 · Abstract (English)
With the rapid expansion in the scale of large language models (LLMs), enabling efficient distributed inference across multiple computing units has become increasingly critical. However, communication overheads from popular distributed inference techniques such as Tensor Parallelism pose a significant challenge to achieve scalability and low latency. Therefore, we introduce a novel optimization technique, Sync-Point Drop (SPD), to reduce communication overheads in tensor parallelism by selectively dropping synchronization on attention outputs. In detail, we first propose a block design that allows execution to proceed without communication through SPD. Second, we apply different SPD strategies to attention blocks based on their sensitivity to the model accuracy. The proposed methods effectively alleviate communication bottlenecks while minimizing accuracy degradation during LLM inference, offering a scalable solution for diverse distributed environments: SPD offered about 20% overall inference latency reduction with < 1% accuracy regression for LLaMA2-70B inference over 8 GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。