通过资源调控实现计算通信重叠，提升多GPU训练效率
Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads
- 用共享内存分配控制计算核驻留，预留通信资源
- 提高通信核优先级，确保资源可用时持续传输
- 不改底层库，在多款GPU上最多提速25.5%
大规模机器学习的快速发展使多GPU分布式训练成为现代机器学习系统的核心。随着模型规模和计算吞吐量不断提升，通信开销已成为多GPU训练的主要瓶颈，尤其在计算与通信串行执行时更为显著。本文探索通过两种可移植的运行时控制机制，实现计算与集合通信的并发执行：利用共享内存驱动的计算核驻留调控，以及对通信核提升调度优先级。方法通过每个线程块的共享内存分配调节计算核的驻留时间，为通信核保留足够的片上资源以推动进展；同时，赋予通信流更高优先级，确保一旦资源可用即可持续推进。在NVIDIA A40、A100、H100及AMD MI250X GPU上的实验表明，该方法能有效实现计算-通信重叠，总执行时间最多降低25.5%，且无需修改厂商库或内核实现。
原文摘要 · Abstract (English)
The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems. As model sizes and computational throughput continue to increase, communication overhead has become a dominant bottleneck in multi-GPU training, particularly when computation and communication are executed sequentially. This work explores concurrent execution of computation and collective communication using two portable runtime controls: shared-memory-driven occupancy shaping for computation kernels and elevated scheduling priority for communication kernels. Our approach regulates computation-kernel residency through per-block shared-memory allocation, leaving sufficient on-chip resources for communication kernels to make progress. In addition, assigning higher priority to communication streams ensures steady communication progress once resources become available. Experiments on NVIDIA A40, A100, H100, and AMD MI250X GPUs demonstrate that the proposed method enables effective computation-communication overlap and reduces total execution time by up to 25.5 percent, without modifying vendor libraries or kernel implementations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。