通过预取专家和推测解码,显著提升受限设备上MoE模型的推理速度。
MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts
- 用小模型预判后续需调用的专家,提前从主机内存预取数据。
- 在内存受限设备上,Phi-MoE模型最高提速2.34倍。
- 适合资源受限场景下部署大规模MoE模型的开发者参考。
当前最先进的混合专家(MoE)模型存在巨大的内存需求,常超出单个加速器的容量。虽然将专家卸载到主机内存是常见解决方案,但专家选择的数据依赖性导致同步传输直接位于执行关键路径上,造成严重的PCIe总线I/O瓶颈,严重拖慢性能。本文提出,可通过少量廉价的设备端计算来隐藏数据移动的巨大开销,从而克服I/O瓶颈。我们设计了MoE-SpeQ,一种基于推测执行与专家卸载协同设计的新推理系统。该系统利用一个小型的设备端草稿模型,预测未来令牌所需专家序列,使运行时调度器能提前从主机内存预取这些专家,有效将昂贵的I/O操作与有用计算重叠,从而隐藏延迟。为最大化性能,一个由损耗屋顶模型引导的自适应调节器,动态调整推测策略以适配底层硬件。在内存受限设备上的评估显示,对于Phi-MoE模型,MoE-SpeQ相比现有最优卸载框架最高实现2.34倍加速。本工作为资源受限环境中的数据依赖内存访问提供了新的、有原则的管理方法,使MoE推理更适用于通用硬件。
原文摘要 · Abstract (English)
The immense memory requirements of state-of-the-art Mixture-of-Experts (MoE) models present a significant challenge for inference, often exceeding the capacity of a single accelerator. While offloading experts to host memory is a common solution, it introduces a severe I/O bottleneck over the PCIe bus, as the data-dependent nature of expert selection places these synchronous transfers directly on the critical path of execution, crippling performance. This paper argues that the I/O bottleneck can be overcome by trading a small amount of cheap, on-device computation to hide the immense cost of data movement. We present MoE-SpeQ, a new inference system built on a novel co-design of speculative execution and expert offloading. MoE-SpeQ employs a small, on-device draft model to predict the sequence of required experts for future tokens. This foresight enables a runtime orchestrator to prefetch these experts from host memory, effectively overlapping the expensive I/O with useful computation and hiding the latency from the critical path. To maximize performance, an adaptive governor, guided by an Amortization Roofline Model, dynamically tunes the speculation strategy to the underlying hardware. Our evaluation on memory-constrained devices shows that for the Phi-MoE model, MoE-SpeQ achieves at most 2.34x speedup over the state-of-the-art offloading framework. Our work establishes a new, principled approach for managing data-dependent memory access in resource-limited environments, making MoE inference more accessible on commodity hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。