用推测解码预判专家需求，提升边缘设备上MoE模型推理效率
MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
- 将推测解码用于预测专家激活，指导内存管理
- 在7个基准上实现42%吞吐提升和4.04倍加速
- 适合资源受限的异构边缘场景部署
混合专家（MoE）模型虽可扩展性能，但在边缘设备上面临严重内存限制。现有卸载策略因自回归专家激活的动态性和低信息性而受制于输入/输出瓶颈。本文提出将推测解码（SD）不仅作为计算加速器，更作为内存管理的前瞻感知器，基于理论与实证分析支持。为此，我们设计了MoE-SpAc框架，包含推测效用评估器以追踪专家需求、异构工作负载均衡器通过在线整数优化动态分配计算，以及异步执行引擎统一预取与驱逐的效用空间。在七个基准上的大量实验表明，相比最先进基于SD的基线，MoE-SpAc实现了42%的TPS提升，且平均比所有标准基线快4.04倍。代码已公开于https://github.com/lshAlgorithm/MoE-SpAc。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models enable scalable performance but face severe memory constraints on edge devices. Existing offloading strategies struggle with I/O bottlenecks due to the dynamic, low-information nature of autoregressive expert activation. In this paper, we propose to repurpose Speculative Decoding (SD) not merely as a compute accelerator, but as an informative lookahead sensor for memory management, supported by our theoretical and empirical analyses. Hence, we introduce MoE-SpAc, an MoE inference framework that integrates a Speculative Utility Estimator to track expert demand, a Heterogeneous Workload Balancer to dynamically partition computation via online integer optimization, and an Asynchronous Execution Engine to unify the prefetching and eviction in the same utility space. Extensive experiments on seven benchmarks demonstrate that MoE-SpAc achieves a 42% improvement in TPS over the SOTA SD-based baseline, and an average 4.04x speedup over all standard baselines. Code is available at https://github.com/lshAlgorithm/MoE-SpAc .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。