优化MoE模型边缘部署，通过智能缓存和预取提升性能。
In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading
- 基于专家激活规律改进缓存策略，用LFU替代传统LRU。
- 实验显示新缓存方案显著降低缺失率，提升推理效率。
- 首次系统分析MoE架构特性，适合研究模型压缩与边缘计算者。
当前，混合专家（MoE）架构被众多先进模型采用，但其内存占用远超密集模型，难以在GPU内存受限的边缘设备上部署。MoE卸载结合缓存与预取技术有望缓解此问题，但现有工作仅停留在次优缓存算法，缺乏深入分析。本文深入研究了MoE卸载，提出三点贡献：1. 详细分析专家激活模式与LRU缓存行为，并提供实际运行轨迹；2. 基于分析提出基于频率的缓存优化（LFU），相比LRU取得显著性能提升；3. 实现并实验了推测性专家预取机制，通过详尽轨迹展示其巨大潜力。此外，本研究全面揭示了MoE架构特性，包括门控网络与专家行为特征，为未来模型解释、剪枝技术开发提供重要参考，支持在极小性能损失下实现高效压缩。
原文摘要 · Abstract (English)
In today's landscape, Mixture of Experts (MoE) is a crucial architecture that has been used by many of the most advanced models. One of the major challenges of MoE models is that they usually require much more memory than their dense counterparts due to their unique architecture, and hence are harder to deploy in environments with limited GPU memory, such as edge devices. MoE offloading is a promising technique proposed to overcome this challenge, especially if it is enhanced with caching and pre-fetching, but prior work stopped at suboptimal caching algorithm and offered limited insights. In this work, we study MoE offloading in depth and make the following contributions: 1. We analyze the expert activation and LRU caching behavior in detail and provide traces. 2. We propose LFU caching optimization based on our analysis and obtain strong improvements from LRU. 3. We implement and experiment speculative expert pre-fetching, providing detailed trace showing its huge potential . 4. In addition, our study extensively covers the behavior of the MoE architecture itself, offering information on the characteristic of the gating network and experts. This can inspire future work on the interpretation of MoE models and the development of pruning techniques for MoE architecture with minimal performance loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。