通过专家替换提升边缘设备上MoE模型的推理效率
SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution
- 用重要性指导专家替换，复用已缓存专家
- 降低48%延迟，专家缓存命中率超60%
- 适合资源受限的边缘部署场景
混合专家（MoE）架构通过仅激活部分专家实现大语言模型的高效扩展。但在消费级边缘硬件上部署时，受限于设备内存，动态专家卸载至关重要。与以往将卸载视为单纯调度问题的做法不同，本文利用专家重要性指导决策，将低重要性激活专家替换为功能相似且已缓存在GPU内存中的专家，从而在保持近似无损精度的同时降低内存占用和数据传输量，显著减少PCIe开销。此外，我们设计了一种最大化GPU缓存专家复用率的调度策略，进一步提升效率。大量实验表明，该方法在保持几乎无损精度的前提下，实现48%的解码延迟降低，专家缓存命中率超过60%。
原文摘要 · Abstract (English)
The Mixture of Experts (MoE) architecture has emerged as a key technique for scaling Large Language Models by activating only a subset of experts per query. Deploying MoE on consumer-grade edge hardware, however, is constrained by limited device memory, making dynamic expert offloading essential. Unlike prior work that treats offloading purely as a scheduling problem, we leverage expert importance to guide decisions, substituting low-importance activated experts with functionally similar ones already cached in GPU memory, thereby preserving accuracy. As a result, this design reduces memory usage and data transfer, while largely eliminating PCIe overhead. In addition, we introduce a scheduling policy that maximizes the reuse ratio of GPU-cached experts, further boosting efficiency. Extensive evaluations show that our approach delivers 48% lower decoding latency with over 60% expert cache hit rate, while maintaining nearly lossless accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。