提出静态批处理不规则任务的框架，提升GPU上MoE模型推理效率。

Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference

• 将不规则任务静态分批并映射到单个核函数执行
• MoE推理核在H800上达峰值张量核心吞吐91%，H20达95%
• 适合需要高效部署MoE模型的工程与研究者

在大规模并行设备上调度和执行不规则工作负载长期存在挑战。本文提出一种通用框架，可将不规则工作负载静态分批为单一内核，并结合运行时任务映射机制在GPU上执行。进一步将该框架应用于混合专家（Mixture-of-Experts, MoE）模型推理，实现优化高效的CUDA内核。所提出的MoE内核在NVIDIA H800 GPU上达到峰值张量核心吞吐的91%，在NVIDIA H20 GPU上达到95%。

原文摘要 · Abstract (English)

It has long been a problem to arrange and execute irregular workloads on massively parallel devices. We propose a general framework for statically batching irregular workloads into a single kernel with a runtime task mapping mechanism on GPUs. We further apply this framework to Mixture-of-Experts (MoE) model inference and implement an optimized and efficient CUDA kernel. Our MoE kernel achieves up to 91% of the peak Tensor Core throughput on NVIDIA H800 GPU and 95% on NVIDIA H20 GPU.

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。