通过预测调度提升多批次MoE模型在普通服务器上的推理效率
LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers
- 基于层感知的激活预测，精准预判专家调用
- 跨层调度使吞吐量提升141%，延迟降低74.6%
- 适合在无专用加速卡的普通服务器上部署MoE模型
混合专家（MoE）模型在通用硬件上部署时面临内存和PCIe延迟瓶颈。将专家权重卸载至CPU内存会导致PCIe传输延迟是GPU计算时间的数倍。我们提出PreScope，一种基于预测的专家调度系统，解决三个关键挑战：激活预测不准、PCIe带宽竞争、跨设备调度复杂性。方案包括：1）可学习的层感知预测器（LLaPor），捕捉各层专家激活模式；2）预取感知的跨层调度（PreSched），生成全局最优计划，平衡预取成本与加载开销；3）异步I/O优化器（AsyncIO），解耦I/O与计算，消除等待空隙。PreScope相比现有最优方案实现141%更高的吞吐量和74.6%更低的延迟。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models face memory and PCIe latency bottlenecks when deployed on commodity hardware. Offloading expert weights to CPU memory results in PCIe transfer latency that exceeds GPU computation by several folds. We present PreScope, a prediction-driven expert scheduling system that addresses three key challenges: inaccurate activation prediction, PCIe bandwidth competition, and cross-device scheduling complexity. Our solution includes: 1) Learnable Layer-Aware Predictor (LLaPor) that captures layer-specific expert activation patterns; 2) Prefetch-Aware Cross-Layer Scheduling (PreSched) that generates globally optimal plans balancing prefetching costs and loading overhead; 3) Asynchronous I/O Optimizer (AsyncIO) that decouples I/O from computation, eliminating waiting bubbles. PreScope achieves 141% higher throughput and 74.6% lower latency than state-of-the-art solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。