揭示MoE调度开销根源,发现现有优化方法存在根本性误判。
Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory

- 构建跨架构观测系统DODOCO,测试五类MoE模型在多种数据条件下的调度表现
- 真实文本下专家负载集中度不随并行度提升而改善,根源在模型路由决策
- 传统模拟数据高估负载不均达2.35倍,且误差为固定偏移而非趋势,适合新硬件设计
AlltoAll调度是MoE专家并行的瓶颈,现有四种缓解方案基于两个假设:路由不平衡可由系统层修正,且模拟令牌基准能准确反映生产路由。本文引入DODOCO,在H100集群上对五类主流MoE模型(MHA、MLA、GQA、Gated DeltaNet、Mamba-2 SSM)进行因子实验,覆盖六种数据条件与专家并行扫描。结果表明两个假设均不成立:并行度提升无法缓解每专家负载集中,长尾问题源于模型自身路由决策;模拟令牌将路由不平衡高估达2.35倍,误差为恒定偏移,不受批大小变化影响;将合成分布转向真实分布(如Zipf)反而扩大差距。进一步发现模型分属两类行为模式:数据鲁棒型(MHA、Mamba-2)在真实文本下路由趋近均匀,而持续集中型(MLA、GDN)保持高负载集中,GQA居中。因此,架构特性而非并行度或模拟数据,才是设计高效互连与调度的关键输入。
原文摘要 · Abstract (English)
AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism, and the interconnect community has responded with four families of mitigations: predictive sample placement, adaptive expert relayout, hierarchical collectives, and EP-aware topology. All four rest on two assumptions about the workload: that routing imbalance is correctable by the system layer, and that the mock-token benchmarks evaluating them faithfully represent production routing. We introduce DODOCO to test both, instrumenting five open MoE checkpoints that span today's sequence-mixer designs (MHA, MLA, GQA, Gated DeltaNet and Mamba-2 SSM) under a factorial grid of six data conditions and a matched expert-parallelism scan on H100 clusters. Both assumptions fail. Scaling EP leaves per-expert load concentration essentially unchanged: the straggler is intrinsic to the routing decision the model makes, not to how its experts land on ranks. Mock tokens overestimate routing imbalance by up to a factor of 2.35, and the error is a level offset rather than a trend: it stays flat across a $32\times$ batch-size sweep. Skewing the synthetic distribution toward realism (Zipf) widens the gap instead of closing it. A third pattern organizes the results: the architectures separate into a data-resilient band (MHA, Mamba-2), whose routing approaches uniform on real text, and a persistently concentrated band (MLA, GDN), with GQA intermediate. These bands, not the EP degree or the mock-data profile, are the right workload input to AlltoAll-aware interconnect and dispatch design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。