通过预测性复制专家，显著提升MoE模型推理速度
Fast MoE Inference via Predictive Prefetching and Expert Replication
- 动态预测高负载专家并提前复制，实现并行处理
- 实测在大模型上达近100%显卡利用率，推理快3倍
- 适合需要高速推理的大型语言模型部署场景
混合专家（MoE）架构已成为顶级大语言模型的核心组件，提升了模型在特定领域的专业能力，并在不显著增加计算开销的前提下扩展模型容量。然而，MoE推理常因专家激活稀疏导致多个令牌等待同一专家计算，从而造成GPU利用率低下、负载不均和延迟升高。为此，我们提出一种动态专家复制策略：预测可能过载的专家，并为其即将到来的令牌批次提前复制。复制后的专家可在各层并发处理批量令牌，提升并行度，减少显卡空闲时间，显著加速推理。在大规模MoE模型（包括Switch-base-128和Switch-base-256）上的实验表明，该方法实现了接近100%的GPU利用率，推理速度最高提升3倍，同时保持基线架构约90%-95%的性能。
原文摘要 · Abstract (English)
The Mixture of Experts (MoE) architecture has become a fundamental building block in state-of-the-art large language models (LLMs), improving domain-specific expertise in LLMs and scaling model capacity without proportionally increasing their computational overhead. However, MoE inference often suffers from suboptimal GPU utilization, load imbalance, and elevated latency arising from multiple tokens waiting on the same experts for their computation which arises from sparsity of expert activation. To address these challenges, we propose a dynamic expert replication strategy that predicts which experts are likely to be overloaded and replicates them for upcoming batches of tokens. The replicated experts process batch tokens concurrently across layers, which leads to improved parallelism, shorter GPU idle time, and significantly faster inference. Experimental evaluations conducted on large-scale MoE models, including Switch-base-128 and Switch-base-256, demonstrate that our method achieves near-complete GPU utilization (approx 100%), leading to upto 3x improvement in inference speed while preserving approximately 90-95% of the performance of baseline architectures
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。