提出动态专家复制的预测策略框架，显著提升MoE模型推理效率。
MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
- 仅预测令牌分布，大幅降低预测开销。
- 在Mixtral 8x7B上实现超过23%的推理性能提升。
- 适合大规模MoE模型部署中的负载均衡优化场景。
在多GPU混合专家（MoE）网络中，专家分布于不同GPU上，导致负载不均，因各专家处理的令牌数量不同。近期工作通过动态复制热门专家到更多GPU以平衡负载，但需预先预测令牌分配。本文探讨预测策略在准确性、开销与端到端系统性能间的权衡。提出MoE-GPS框架，通过量化预测策略对系统运行时的影响，指导在不同配置下选择最优预测设计。特别倡导仅预测整体令牌分布的「Distribution-Only Prediction」策略，相比传统的「Token-to-Expert Prediction」显著降低开销。在Mixtral 8x7B和MMLU数据集上，该策略使端到端推理性能提升超过23%。
原文摘要 · Abstract (English)
In multi-GPU Mixture-of-Experts (MoE) network, experts are distributed across different GPUs, which creates load imbalance as each expert processes different number of tokens. Recent works improve MoE inference load balance by dynamically duplicating popular experts to more GPUs to process excessive tokens, which requires predicting the distribution before routing. In this paper, we discuss the tradeoff of prediction strategies, accuracies, overhead, and end-to-end system performance. We propose MoE-GPS, a framework that guides the selection of the optimal predictor design under various system configurations, by quantifying the performance impact to system-level model runtime. Specifically, we advocate for Distribution-Only Prediction, a prediction strategy that only predicts overall token distribution which significantly reduces overhead compared to the traditional Token-to-Expert Prediction. On Mixtral 8x7B MMLU dataset, MoE-GPS suggests Distribution-Only Prediction which improves end-to-end inference performance by more than 23% compared with Token-to-Expert Prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。