TEMPO动态调度专家并行负载，提升大模型推理吞吐与延迟表现。
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
- 基于分段线性模型动态识别计算与内存瓶颈，智能分配专家任务。
- 在混合瓶颈场景下，实测吞吐提升4%-6%，尾延迟降低15.6%。
- 适合多专家模型推理部署，尤其适用于复杂硬件环境下的性能优化。
在专家并行（EP）MoE服务中，每一层的同步受最慢GPU制约。现有调度器基于令牌数或激活专家数平衡负载（如EPLB、LPLB、UltraEP、METRO），但假设专家耗时与输入成线性关系——实测表明该假设不成立：当输入低于约156–168个令牌时，显存权重流控主导，成本关联于激活副本数；超过此值后，分组GEMM将令牌合并为128-块的M块，拆分专家会引入填充计算开销。一个最大仿射模型 $t = \max(a + bG, c + βN)$ 可同时刻画两阶段行为。实际解码批次中热专家处于线性区，冷专家处于平缓区，且二者共存；记录数据显示，不同代理调度的区块时间差异达1.4–1.6倍（p95高达1.7倍），哪个代理更优随阶段切换而变化。本文将每批调度形式化为固定费用的完成时间最小化问题——在两台完全复制的GPU上为NP难，但在退化极限下可多项式求解，并提出TEMPO调度器，在关键路径外毫秒级求解；其集成至SGLang后以图内核融合调度与计数收集。基于8卡测试平台A的微基准测试显示，TEMPO始终接近最优固定基线（误差<1%），在混合阶段最高领先15.5%。端到端测试平台B上，Qwen3-235B（位于优势区域）实现4–6%吞吐提升，尾延迟下降约15.6%；DeepSeek-V3（外部，通信主导）仅体现机制开销。核心结论非普适性胜利，而是通过相图预测部署前的表现结果。
原文摘要 · Abstract (English)
In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU. Dispatchers balance token counts (EPLB, LPLB, UltraEP) or activated-expert counts (METRO), assuming expert time is linear in one. Measurements on two datacenter GPU generations show it is neither: below $n^* \approx 156$--$168$ tokens, HBM weight streaming dominates---cost attaches to $activated replicas$, not tokens; above it, grouped GEMM rounds tokens to 128-tile $M$-tiles, so $splitting$ an expert adds padded compute. A max-affine profile $t=\max(a+bG,\,c+βN)$ captures both regimes. Realistic decode batches hold hot experts in the linear regime and cold in the flat $simultaneously$; recorded batches show proxy dispatches differ by $1.4$--$1.6\times$ in modeled block time (p95 up to $1.7\times$), and $which$ proxy wins flips with the regime. We formalize per-batch dispatch as a fixed-charge makespan problem---NP-hard on two fully replicated GPUs, polynomial in degenerate limits---and present TEMPO, a makespan-aware dispatcher solving it in milliseconds off the critical path; its SGLang integration runs out-of-process and fuses dispatch with count collection into one in-graph kernel. Anchored by an 8-GPU Testbed A microbenchmark, TEMPO stays within $1\%$ of the best fixed baseline everywhere and wins by up to $15.5\%$ where regimes mix. End-to-end on Testbed B, Qwen3-235B (inside the win region) gains $4$--$6\%$ throughput and cuts p99 latency by $\sim 15.6\%$; DeepSeek-V3 (outside, communication-dominated) shows only mechanism cost. A phase diagram, not a universal win, is the claim: it predicts both outcomes before deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。