通过自助式推测解码，让专家模型推理更快更省内存。
SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
- 用自助推测解码优化专家模型推理流程，无需额外训练。
- 在内存受限系统上，推理吞吐最高提升4.3倍，带宽需求大幅降低。
- 适合需要高效部署大规模专家模型的研究者和工程团队。
混合专家（MoE）架构通过选择性激活参数，为缓解大语言模型日益增长的计算成本提供了有前景的解决方案。然而，其高内存需求和参数效率不佳给高效部署带来了显著挑战。尽管已有研究提出将MoE推理卸载到CPU的系统，但对大批次场景的效率仍有限。本文提出SpecMoE，一种基于自助式推测解码算法的内存高效的MoE推理系统。该系统在无需额外模型训练或微调的前提下，成功将推测解码应用于MoE推理。实验表明，SpecMoE在内存受限系统上，推理吞吐最高提升4.30倍，同时显著降低了内存与互连带宽的需求。
原文摘要 · Abstract (English)
The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs) by selectively activating parameters. However, its high memory requirements and sub-optimal parameter efficiency pose significant challenges for efficient deployment. Although CPU-offloaded MoE inference systems have been proposed in the literature, they offer limited efficiency, particularly for large batch sizes. In this work, we propose SpecMoE, a memory-efficient MoE inference system based on our self-assisted speculative decoding algorithm. SpecMoE demonstrates the effectiveness of applying speculative decoding to MoE inference without requiring additional model training or fine-tuning. Our system improves inference throughput by up to $4.30\times$, while significantly reducing bandwidth requirements of both memory and interconnect on memory-constrained systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。