通过提前预测专家使用，提升MoE大模型推理速度
ProMoE: Fast MoE-based LLM Serving using Proactive Caching
- 利用中间结果预判后续专家调用，提前缓存
- 预填充阶段平均快2.20倍，解码阶段快2.07倍
- 适合部署在显存受限的边缘设备上
大语言模型的应用常受边缘设备显存容量限制。混合专家（MoE）模型通过仅激活部分参数来缓解此问题，未使用的参数可卸载至主机内存，降低整体显存需求。然而，现有基于缓存的卸载方案采用被动处理缓存未命中，严重影响系统性能。本文提出ProMoE，一种新型主动缓存系统，利用中间结果预测后续专家使用，提前加载专家。该方法消除被动缓存未命中，将加载时间移出关键路径，减少卸载带来的性能开销。评估表明，与现有方案相比，ProMoE在预填充阶段平均提速2.20倍（最高3.21倍），解码阶段平均提速2.07倍（最高5.02倍）。
原文摘要 · Abstract (English)
The promising applications of large language models are often limited by the constrained GPU memory capacity available on edge devices. Mixture-of-Experts (MoE) models help address this issue by activating only a subset of the model's parameters during computation. This approach allows the unused parameters to be offloaded to host memory, thereby reducing the overall GPU memory demand. However, existing cache-based offloading solutions handle cache misses reactively, which significantly impacts system performance. In this paper, we introduce ProMoE, a novel proactive caching system that utilizes intermediate results to predict subsequent expert usage. By proactively fetching experts in advance, ProMoE eliminates passive cache misses, removes loading time from the critical path, and reduces the performance overhead associated with offloading. Our evaluations demonstrate that ProMoE achieves an average speedup of 2.20x (up to 3.21x) and 2.07x (up to 5.02x) in the prefill and decode stages, respectively, compared to existing offloading solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。