让苹果芯片高效运行专家混合模型，显著降低延迟和功耗。
Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs
- 将静态计算卸载到NPU，动态操作保留CPU/GPU备用路径。
- 通过离线校准优化专家容量与负载，减少调度开销。
- 适合在苹果M系列芯片上部署长上下文专家模型的开发者。
Apple Neural Engine（ANE）是每款Apple Silicon芯片中的专用神经网络处理单元（NPU）。专家混合（MoE）大语言模型通过稀疏激活提升推理效率，但对NPU带来三大挑战：专家路由不可预测，引入动态张量形状，与NPU的定长形状约束冲突；多个不规则操作（如top-k、scatter/gather）不兼容NPU；频繁启动小型专家内核导致显著的调度与同步开销。我们的目标是实现MoE推理在NPU上的有效卸载，尤其在预填充阶段，该阶段长上下文任务消耗大量系统资源。本文提出NPUMoE，一种运行时推理引擎，通过将密集静态计算卸载至NPU，同时保留CPU/GPU回退路径以应对动态操作。NPUMoE利用离线校准估算专家容量与流行度，驱动三项关键技术：（1）静态分层设计以应对动态专家路由；（2）分组专家执行以缓解NPU并发限制；（3）基于负载感知的专家计算图驻留策略，减少CPU-NPU同步开销。在Apple M系列设备上，使用三个代表性MoE LLM和四个长上下文工作负载的实验表明，NPUMoE持续优于基线，延迟降低1.32x–5.55x，能效提升1.81x–7.37x，CPU周期使用减少1.78x–5.54x，实现高效NPU卸载。
原文摘要 · Abstract (English)
Apple Neural Engine (ANE) is a dedicated neural processing unit (NPU) present in every Apple Silicon chip. Mixture-of-Experts (MoE) LLMs improve inference efficiency via sparse activation but are challenging for NPUs in three ways: expert routing is unpredictable and introduces dynamic tensor shapes that conflict with the shape-specific constraints of NPUs; several irregular operators, e.g., top-k, scatter/gather, etc., are not NPU-friendly; and launching many small expert kernels incurs substantial dispatch and synchronization overhead. NPUs are designed to offload AI compute from CPU and GPU; our goal is to enable such offloading for MoE inference, particularly during prefill, where long-context workloads consume substantial system resources. This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon by offloading dense, static computation to NPU, while preserving a CPU/GPU fallback path for dynamic operations. NPUMoE uses offline calibration to estimate expert capacity and popularity that drives three key techniques: (1) Static tiers for expert capacity to address dynamic expert routing (2) Grouped expert execution to mitigate NPU concurrency limits (3) Load-aware expert compute graph residency to reduce CPU-NPU synchronization overhead. Experiments on Apple M-series devices using three representative MoE LLMs and four long-context workloads show that NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。