分析大模型专家选择的数据移动规律,提升推理效率。
Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference
- 通过大规模实测发现专家选择存在可预测模式。
- 基于洞察在晶圆级GPU上实现6.6倍平均加速。
- 适合关注大模型高效推理的系统研发人员。
大规模混合专家(MoE)大语言模型已成为前沿开源模型,性能接近闭源模型。但其随机专家选择机制导致显著的数据移动开销,成为多设备部署系统中的主要瓶颈。为理解这一数据移动背后的规律,我们在2025年发布的四款先进大尺度MoE模型(200B–1000B)上,基于超过24,000个请求、涵盖多样工作负载进行了全面的数据移动中心型剖析。从时序与空间双角度系统分析,提炼出六项关键洞见,指导多样化服务系统设计。这些洞见在未来的晶圆级GPU架构和现有GPU系统上均得到验证:在晶圆级GPU上,轻量级架构优化带来四款模型平均6.6×加速;在现有系统中,基于预填充感知的专家放置算法可实现高达1.25×的MoE计算加速。本工作首次提供大规模MoE模型的全数据驱动分析,并完成可落地的设计验证。分析轨迹已公开于Hugging Face:https://huggingface.co/datasets/core12345/MoE_expert_selection_trace。
原文摘要 · Abstract (English)
Large-scale Mixture of Experts (MoE) Large Language Models (LLMs) have recently become the frontier open-weight models, achieving remarkable model capability similar to proprietary ones. But their random expert selection mechanism introduces significant data movement overhead that becomes the dominant bottleneck in multi-unit LLM serving systems. To understand the patterns underlying this data movement, we conduct comprehensive data-movement-centric profiling across four state-of-the-art large-scale MoE models released in 2025 (200B-1000B) using over 24,000 requests spanning diverse workloads. We perform systematic analysis from both temporal and spatial perspectives and distill six key insights to guide the design of diverse serving systems. We verify these insights on both future wafer-scale GPU architectures and existing GPU systems. On wafer-scale GPUs, lightweight architectural modifications guided by our insights yield a 6.6$\times$ average speedup across four 200B--1000B models. On existing GPU systems, our insights drive the design of a prefill-aware expert placement algorithm that achieves up to 1.25$\times$ speedup on MoE computation. Our work presents the first comprehensive data-centric analysis of large-scale MoE models together with a concrete design study applying the learned lessons. Our profiling traces are publicly available at \href{https://huggingface.co/datasets/core12345/MoE_expert_selection_trace}{\textcolor{blue}{https://huggingface.co/datasets/core12345/MoE\_expert\_selection\_trace}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。