通过利用专家冗余加速受限内存下的MoE推理
BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
- 利用专家间的冗余性，在预取失败时仍能快速恢复计算
- 在混合专家模型中仅激活部分专家，降低内存占用至140亿参数/令牌
- 适合需要高速低延迟推理的部署场景，如实时对话系统
混合专家（MoE）架构通过为每个输入令牌仅激活部分专用专家网络来扩展语言模型，从而减少浮点运算量。然而，现代MoE模型规模不断增大，其完整参数集已超出GPU内存容量；例如，Mixtral-8x7B拥有450亿参数，即使每令牌仅使用140亿参数，也需87GB内存。现有系统通过将未激活专家卸载至CPU内存缓解此问题，但跨PCIe总线传输专家会引入显著延迟（约10毫秒）。预取启发式方法试图通过预测所需专家来隐藏延迟，但预取失败会导致严重停顿并加剧推理延迟。预取失败时，先前工作主要提供两种方案：按需获取专家，因PCIe瓶颈导致长时间停顿；或直接丢弃专家，显著降低模型准确率。因此，核心挑战在于预取失败时同时保持高推理速度和模型准确性。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures scale language models by activating only a subset of specialized expert networks for each input token, thereby reducing the number of floating-point operations. However, the growing size of modern MoE models causes their full parameter sets to exceed GPU memory capacity; for example, Mixtral-8x7B has 45 billion parameters and requires 87 GB of memory even though only 14 billion parameters are used per token. Existing systems alleviate this limitation by offloading inactive experts to CPU memory, but transferring experts across the PCIe interconnect incurs significant latency (about 10 ms). Prefetching heuristics aim to hide this latency by predicting which experts are needed, but prefetch failures introduce significant stalls and amplify inference latency. In the event of a prefetch failure, prior work offers two primary solutions: either fetch the expert on demand, which incurs a long stall due to the PCIe bottleneck, or drop the expert from the computation, which significantly degrades model accuracy. The critical challenge, therefore, is to maintain both high inference speed and model accuracy when prefetching fails.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。