eMoE通过智能加载专家模型，大幅降低内存占用和推理延迟。
eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference
- 仅按需加载所需专家，基于路由模式预测减少内存占用
- 每几轮才预测一次，关键任务跳过预测，降低延迟
- 根据任务特点动态调度，适合高并发、长文本推理场景
近年来，混合专家（MoE）已成为提升深度神经网络容量且计算成本近似线性的有效方法。然而，将所有专家存储在GPU上会带来显著的内存开销，增加基于MoE的推理成本。为解决此问题，我们提出eMoE，一种面向大语言模型的内存高效推理系统。该系统基于实验观测，通过预测并仅加载所需专家来减少内存使用。由于连续提示中重复使用相同专家对困惑度影响极小，eMoE每隔若干提示才调用专家预测器，而非每条提示都执行。此外，对路由精度不敏感的任务跳过预测。最后，结合服务等级目标（SLO）、任务输出长度和专家加载延迟，实现任务感知调度以最小化推理延迟。实验表明，相比现有系统，eMoE可降低高达80%的内存消耗，同时保持精度，并减少17%的推理延迟；还能处理40倍更长的提示、4.5倍更大的批量，实现1.5倍更高的吞吐量。
原文摘要 · Abstract (English)
In recent years, Mixture-of-Experts (MoE) has emerged as an effective approach for enhancing the capacity of deep neural network (DNN) with sub-linear computational costs. However, storing all experts on GPUs incurs significant memory overhead, increasing the monetary cost of MoE-based inference. To address this, we propose eMoE, a memory efficient inference system for MoE-based large language models (LLMs) by leveraging our observations from experiment measurements. eMoE reduces memory usage by predicting and loading only the required experts based on recurrent patterns in expert routing. To reduce loading latency while maintaining accuracy, as we found using the same experts for subsequent prompts has minimal impact on perplexity, eMoE invokes the expert predictor every few prompts rather than for each prompt. In addition, it skips predictions for tasks less sensitive to routing accuracy. Finally, it has task-aware scheduling to minimize inference latency by considering Service Level Objectives (SLOs), task-specific output lengths, and expert loading latencies. Experimental results show that compared to existing systems, eMoE reduces memory consumption by up to 80% while maintaining accuracy and reduces inference latency by up to 17%. It also enables processing prompts 40x longer, batches 4.5x larger, and achieves 1.5x higher throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。