提出新推理并行策略，提升多GPU大模型推理效率。
DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
- 去同步化设计，各GPU独立执行不等待
- 在8000输入+1000输出下，每卡吞吐提升8.8%
- 适合高并发大模型服务场景，尤其多GPU部署
大语言模型推理日益依赖多GPU执行，但现有并行策略需层间跨卡同步，导致整体性能受负载不均影响。本文提出分布式权重数据并行（DWDP），在保持数据并行性的同时，将MoE模型的专家权重分布到同级GPU上，并按需加载缺失专家。通过消除集体跨卡同步，使每个GPU可独立推进。进一步通过两项优化解决分片权重管理与异步远程权重预取的开销。在TensorRT-LLM中实现，基于DeepSeek-R1在GB200 NVL72上评估，当输入序列长度为8K、输出为1K时，在20–100 TPS/user范围内，每卡端到端输出吞吐率提升8.8%，且用户吞吐基本持平。
原文摘要 · Abstract (English)
Large language model (LLM) inference increasingly depends on multi-GPU execution, yet existing inference parallelization strategies require layer-wise inter-rank synchronization, making end-to-end performance sensitive to workload imbalance. We present DWDP (Distributed Weight Data Parallelism), an inference parallelization strategy that preserves data-parallel execution while offloading MoE weights across peer GPUs and fetching missing experts on demand. By removing collective inter-rank synchronization, DWDP allows each GPU to progress independently. We further address the practical overheads of this design with two optimizations for split-weight management and asynchronous remote-weight prefetch. Implemented in TensorRT-LLM and evaluated with DeepSeek-R1 on GB200 NVL72, DWDP improves end-to-end output TPS/GPU by 8.8% at comparable TPS/user in the 20-100 TPS/user serving range under 8K input sequence length and 1K output sequence length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。