通过细粒度专家调度，显著降低MoE模型推理延迟并提升内存效率。
Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
- 基于输入提示和专家激活模式，动态精细调度专家的预加载与卸载。
- 在真实负载下，推理延迟降低47%，专家命中率提升39%。
- 适合追求高并发、低延迟部署的MoE大模型服务场景。
大型语言模型（LLM）在内容生成、搜索推荐和AI辅助运营等领域取得巨大成功。为降低训练成本，混合专家（MoE）架构已成为现代LLM的主流设计。然而，由于专家稀疏激活，MoE模型在服务时面临严重内存效率问题。现有方法将未激活专家从GPU卸载至CPU以提升效率，但普遍存在推理延迟高或内存开销大的问题。为此，本文提出FineMoE，一种细粒度专家卸载系统，可在保证低延迟的同时实现内存高效。FineMoE通过提取MoE模型中的细粒度专家选择模式及输入提示的语义线索，智能指导专家的预取、缓存与卸载决策。系统基于HuggingFace Transformers原型，并部署于六卡测试环境。在开源MoE模型与真实工作负载上的实验表明，相比当前最优方案，FineMoE可降低47%推理延迟，并将专家命中率提升39%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have gained immense success in revolutionizing various applications, including content generation, search and recommendation, and AI-assisted operation. To reduce high training costs, Mixture-of-Experts (MoE) architecture has become a popular backbone for modern LLMs. However, despite the benefits, serving MoE-based LLMs experience severe memory inefficiency due to sparsely activated experts. Recent studies propose to offload inactive experts from GPU memory to CPU memory to improve the serving efficiency of MoE models. However, they either incur high inference latency or high model memory footprints due to coarse-grained designs. To tame the latency-memory trade-off in MoE serving, we present FineMoE, a fine-grained expert offloading system for MoE serving that achieves low inference latency with memory efficiency. We design FineMoE to extract fine-grained expert selection patterns from MoE models and semantic hints from input prompts to efficiently guide expert prefetching, caching, and offloading decisions. FineMoE is prototyped on top of HuggingFace Transformers and deployed on a six-GPU testbed. Experiments with open-source MoE models and real-world workloads show that FineMoE reduces inference latency by 47% and improves expert hit rate by 39% over state-of-the-art solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。