通过线程组流水线提升多GPU系统延迟性能，适合大模型低延迟部署。
CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems
- 利用线程组依赖关系实现跨GPU任务流水执行
- 8卡H200/B200上2层GEMM延迟降低31.8%（相比微批处理）
- 可与张量并行结合，突破现有延迟瓶颈
计算基础设施的演进使多GPU系统逐渐成为紧密集成的共享内存结构，但当前软件仍将其视为高速网络。随着大模型在延迟约束下的服务需求增长，GPU工作负载优化正从吞吐驱动转向延迟敏感，亟需超越张量并行的延迟导向扩展方法。为此，我们提出CTA-pipelining，一种面向共享内存多GPU系统的执行范式。该方法基于协作线程数组（Cooperative Thread Array, CTA）级别的依赖关系，实现跨GPU的依赖核函数并发执行。我们在8卡H200和B200系统上使用CUTLASS、cuBLAS和NCCL库验证了其能力。结果表明，在代表MLP操作的2层GEMM上，与微批处理相比，延迟最高降低31.8%；与张量并行相比，降低29.6%。该方法还可作为与张量并行正交的扩展维度，进一步压缩延迟边界。
原文摘要 · Abstract (English)
The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures. However, current software still mostly treats these coherent interconnects simply as high-speed networks. Simultaneously, the demand for serving Large Language Models under latency constraints has shifted GPU workload optimization from being throughput-driven to latency-bound, necessitating latency-oriented scaling methods beyond Tensor Parallelism (TP). Thus, we introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems. As a latency-oriented spatial scaling technique, CTA-pipelining leverages dependencies at the Cooperative Thread Array level, enabling concurrent execution of dependent kernels across GPUs. We demonstrate its capability using CUTLASS, cuBLAS, and NCCL libraries on 8-GPU H200 and B200 systems. Results show on 2-layer GEMM, representing the MLP operation, CTA-pipelining reduces latency by up to 31.8% compared to micro-batching, and 29.6% compared to TP. It can also be combined with TP as an orthogonal scaling dimension to further push the latency boundary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。