用无服务器架构解决MoE模型推理负载不均问题,显著降低延迟和成本。
MoEless: Efficient MoE LLM Serving via Serverless Computing
- 通过轻量级预测器提前识别专家负载与瓶颈
- 在8卡测试环境中实现43%延迟下降、84%成本降低
- 适合高并发、弹性需求的MoE模型部署场景
大型语言模型(LLM)已成为AI的核心,推动内容生成、搜索推荐及AI辅助工作流的发展。为缓解训练成本并提升模型规模,现代LLM广泛采用专家混合(MoE)架构,通常通过专家并行(EP)分布式部署。然而,MoE的稀疏激活机制导致专家负载严重不均:少数专家过载而其他专家闲置,引发专家拖尾现象,增加推理延迟与服务成本。现有负载均衡方案依赖静态资源配置,限制了专家的可扩展性与弹性,导致实时专家切换代价高昂或生成质量下降。本文提出MoEless,首个基于无服务器架构的MoE推理框架,通过轻量级、分层感知的预测器准确估算输入专家负载分布,并主动识别潜在瓶颈。设计优化的专家弹性伸缩与部署策略,最大化函数局部性,提升GPU利用率并平衡专家与GPU间负载。在基于Megatron-LM的原型系统上,于8卡测试环境部署。实验使用开源MoE模型与真实工作负载,结果表明相比现有最佳方案,MoEless将推理延迟降低43%,成本减少84%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become a cornerstone of AI, driving progress across diverse domains such as content creation, search and recommendation systems, and AI-assisted workflows. To alleviate extreme training costs and advancing model scales, Mixture-of-Experts (MoE) has become a popular backbone for modern LLMs, which are commonly served in distributed deployment using expert parallelism (EP). However, MoE's sparse activation mechanism leads to severe expert load imbalance, where a few experts become overloaded while others remain idle, resulting in expert stragglers that inflate inference latency and serving cost. Existing expert load balancing solutions assume static resource configurations on serverful infrastructures, limiting expert scalability and elasticity, and resulting in either costly real-time expert swapping or degraded generation quality. We present MoEless, the first serverless MoE serving framework that mitigates expert load imbalance and accelerates inference via serverless experts. MoEless employs lightweight, layer-aware predictors to accurately estimate incoming expert load distributions and proactively identify stragglers. We design optimized expert scaling and placement strategies to maximize function locality, improve GPU utilization, and balance loads across experts and GPUs. MoEless is prototyped on top of Megatron-LM and deployed on an eight-GPU testbed. Experiments with open-source MoE models and real-world workloads show that MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。