用学习方法预测专家激活，提升边缘设备上MoE模型的缓存效率
MoE-Beyond: Learning-Based Expert Activation Prediction on Edge Devices
- 训练轻量Transformer预测推理时专家激活情况
- 在仅10%专家可缓存时，缓存命中率从17%升至72%
- 适合部署大规模MoE模型的边缘设备开发者
在边缘设备上部署大规模混合专家（MoE）模型面临内存瓶颈。尽管MoE通过每轮仅激活部分专家实现计算高效，但在资源受限环境下仍需精细内存管理。传统基于启发式的专家缓存策略（如MoE-Infinity）随模型参数规模扩大，难以维持高缓存命中率。本文提出MoE-Beyond，一种基于学习的专家激活预测器，将任务建模为多标签序列预测问题，在6600万条来自LDJnr-Puffin数据集的专家激活轨迹上，使用DeepSeek-V2-Chat-Lite MoE训练轻量Transformer模型。该预测器在WebGLM-QA未见提示上表现良好，准确率达97.5%，F1得分为86.6%。仿真结果显示，当仅10%专家能装入GPU缓存时，MoE-Beyond将缓存命中率从17%提升至72%，显著优于启发式基线。
原文摘要 · Abstract (English)
The deployment of large-scale Mixture-of-Experts (MoE) models on edge devices presents significant challenges due to memory constraints. While MoE architectures enable efficient utilization of computational resources by activating only a subset of experts per inference, they require careful memory management to operate efficiently in resource-constrained environments. Traditional heuristic-based expert caching strategies such as MoE-Infinity struggle to maintain high cache hit rates as models parameters scale. In this work, we introduce MoE-Beyond, a learning-based expert activation predictor trained to predict expert activations during autoregressive decoding. By framing the task as a multi-label sequence prediction problem, we train a lightweight transformer model on 66 million expert activation traces extracted from LDJnr-Puffin dataset [5] using DeepSeek-V2-Chat-Lite MoE. Our predictor generalizes effectively across unseen prompts from WebGLM-QA dataset [6], achieving 97.5% accuracy and an 86.6% F1-score. Simulation results show that MoE-Beyond improves GPU cache hit rate from 17% to 72% when only 10% of experts fit in GPU cache, outperforming heuristic baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。