通过智能预取专家模型,显著降低边缘设备推理延迟和能耗。
APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

- 根据置信度动态预测并预加载专家,避免内存瓶颈
- 正确性保留模式下延迟降低26%,能效比提升41%
- 适合资源受限的边缘设备部署大规模MoE模型
混合专家(MoE)模型因其高容量与低激活参数量,适用于边缘部署。然而,边缘推理受内存限制:专家参数大且常存于片外内存,加载成为关键路径瓶颈。本文提出APEX:自适应专家预取框架,通过轻量级预取路由器,在注意力层前预测候选专家,并基于学习的置信度模型动态加载。该策略实现超过99%的重叠准确率,显著优于固定top-k预取。APEX支持两种模式:保持正确性的模式保证路由语义精确,减少每标记延迟达26%,能效延迟积(EDP)提升41%;无停顿模式通过利用可用专家消除残留停顿,几乎不影响精度。实验表明,自适应、置信度驱动的预取是边缘高效MoE推理的有效方案。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency. However, MoE inference at the edge is fundamentally limited by memory. Expert parameters are large and often reside in off-chip memory due to capacity, cost, and power constraints, putting expert loading to the critical path. We present APEX: Adaptive Expert Prefetching, a predictive resource management framework that overlaps expert loading with useful computation. APEX introduces a lightweight prefetch router that predicts candidate experts before the attention block to dynamically fetch additional experts using a learned confidence model. This adaptive strategy achieves over 99% overlap accuracy, significantly outperforming fixed top-k prefetching techniques. APEX supports two execution modes: a correctness-preserving mode that guarantees exact routing semantics, and a stall-free mode that eliminates residual stalls by operating on available experts with negligible impact on application accuracy. Across multiple MoE models, the correctness-preserving mode reduces per-token latency by up to 26% and improves energy-delay product (EDP) by up to 41% over state-of-the-art baselines, while the stall-free mode provides additional efficiency gains with negligible impact on application accuracy. These results establish adaptive, confidence-driven expert prefetching as an effective approach for efficient MoE inference on edge systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。