通过缓存感知路由提升移动端MoE推理速度，实现2倍加速。
Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
- 设计缓存感知路由，利用生成过程中的专家复用提升缓存命中率。
- 在移动端实测语言建模、MMLU和GSM8K任务中提速2倍。
- 无需训练即可部署，适合资源受限的移动设备应用。
混合专家（MoE）大模型通过为每个输入选择性激活专用子网络，显著提升性能。然而，在内存受限设备上部署MoE仍具挑战性，尤其在单批次、逐个令牌生成的场景下，与高吞吐的长序列或大批次设置不同。本文针对仅部分专家权重可放入DRAM的移动端环境，提出一种新的缓存感知路由策略，利用生成过程中专家复用以增强缓存局部性。我们在语言建模、MMLU和GSM8K基准上评估该方法，展示在真实移动设备上的实验结果：推理速度提升2倍。该方案无需训练，具有良好的灵活性，可广泛应用于实际移动端场景。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) LLMs have recently gained attention for their ability to enhance performance by selectively engaging specialized subnetworks or "experts" for each input. However, deploying MoEs on memory-constrained devices remains challenging, particularly when generating tokens sequentially with a batch size of one, as opposed to typical high-throughput settings involving long sequences or large batches. In this work, we optimize MoE on memory-constrained devices where only a subset of expert weights fit in DRAM. We introduce a novel cache-aware routing strategy that leverages expert reuse during token generation to improve cache locality. We evaluate our approach on language modeling, MMLU, and GSM8K benchmarks and present on-device results demonstrating 2$\times$ speedups on mobile devices, offering a flexible, training-free solution to extend MoE's applicability across real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。