评测MoE模型在不同场景下的推理性能，助力高效部署。
MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
- 构建多场景测试基准，分析批量大小、序列长度等影响。
- 实测显示不同专家数量与前馈层维度显著影响吞吐量。
- 适合研究者与工程师优化MoE模型的硬件加速方案。
混合专家（MoE）模型通过大规模参数量实现大语言模型（LLMs）和视觉语言模型（VLMs）的扩展，同时保持计算效率。然而，MoE引入了推理阶段的负载不均及路由计算开销等挑战。为充分释放其潜力，需系统评估硬件加速技术。本文提出MoE-Inference-Bench，对多种场景下的MoE性能进行全面评估。分析了批处理大小、序列长度及关键超参数（如前馈网络维度、专家数量）对吞吐量的影响。在Nvidia H100 GPU上测试了剪枝、融合MoE操作、推测解码、量化及多种并行策略。涵盖Mixtral、DeepSeek、OLMoE和Qwen系列模型。结果揭示了不同配置间的性能差异，为MoE的高效部署提供了实践指导。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) models have enabled the scaling of Large Language Models (LLMs) and Vision Language Models (VLMs) by achieving massive parameter counts while maintaining computational efficiency. However, MoEs introduce several inference-time challenges, including load imbalance across experts and the additional routing computational overhead. To address these challenges and fully harness the benefits of MoE, a systematic evaluation of hardware acceleration techniques is essential. We present MoE-Inference-Bench, a comprehensive study to evaluate MoE performance across diverse scenarios. We analyze the impact of batch size, sequence length, and critical MoE hyperparameters such as FFN dimensions and number of experts on throughput. We evaluate several optimization techniques on Nvidia H100 GPUs, including pruning, Fused MoE operations, speculative decoding, quantization, and various parallelization strategies. Our evaluation includes MoEs from the Mixtral, DeepSeek, OLMoE and Qwen families. The results reveal performance differences across configurations and provide insights for the efficient deployment of MoEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。