突破传统分布式训练瓶颈，实现多GPU大模型更高效执行。
Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs
- 提出细粒度数据流编程模式，替代僵化的同步计算框架。
- 在关键算子上实现10%-20%端到端延迟降低，提升训练效率。
- 适合关注分布式大模型性能优化的研究者与工程实践者。
随着大语言模型（LLMs）持续扩大规模，其计算任务越来越多依赖于多GPU分布式执行。然而，传统的批量同步并行（BSP）模型在此类场景中引入了显著的性能损耗。为系统刻画这些瓶颈，我们提出‘三税’分析框架——批量同步税、核间数据局部性税、核启动开销税。通过采用Iris等Triton库提供的内核级通信原语，设计新型细粒度编程模式，摆脱对传统BSP的依赖。该方法通过构建逐块级的生产者-消费者流水线，将全局屏障替换为细粒度数据流同步，系统性地消除了三类性能损耗。将该方法应用于核心算子，包括基础的All-Gather + 通用矩阵乘法以及复杂的Flash Decode算法，在端到端延迟上相较BSP方案提升10%-20%，建立了一种更具可编程性与效率的分布式LLM执行范式。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to scale, their workloads increasingly rely on distributed execution across multiple GPUs. However, the conventional bulk synchronous parallel~(BSP) model used in such settings introduces significant performance inefficiencies. To characterize these bottlenecks, we introduce the ''Three Taxes'' (Bulk Synchronous, Inter-Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework. We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution. By exploiting libraries like Iris for Triton, we gain access to in-kernel communication primitives that enable the design of novel fine-grained programming patterns, offering greater flexibility and performance than traditional BSP-based approaches. These patterns systematically eliminate the three taxes by creating direct, tile-level producer-consumer pipelines and replacing global barriers with fine-grained dataflow synchronization. Applying this methodology to critical kernels, from the foundational All-Gather + general matrix multiplication operation to the complex Flash Decode algorithm, we observe a 10-20% speedup in end-to-end latency over BSP-based approaches, establishing a more programmable and efficient paradigm for distributed LLM workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。