arXiv:2606.21428cs.PFcs.AI2026-06

MoE模型在边缘设备上未必省力省电,实际性能受内存和带宽限制。

Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

  • 对比稀疏激活的MoE与密集模型,测试其在消费级硬件上的真实推理表现。
  • 在Jetson Orin上,MoE模型比同规模稠密模型慢31%,能耗高2.1倍,内存达8GB上限。
  • 瓶颈不在路由计算,而是总参数量带来的内存占用和缓存压力。

混合专家(MoE)语言模型常被认为适合资源受限的推理场景。每个标记仅激活少数专家,理论上的每标记计算量(FLOPs)接近更小的稠密模型。但这种FLOP优势在实践中是否成立仍不明确。我们通过llama.cpp在Apple M2 Pro和NVIDIA Jetson Orin Nano 8GB上对OLMoE-1B-7B(13亿活跃参数,总计69亿参数)与三个稠密基线进行基准测试,评估吞吐量、内存占用及设备能耗。结果表明,性能依赖于设备:在笔记本上,其表现落后于相同活跃参数的Llama-3.2-1B约10%;在边缘设备上则落后31%,单位标记能耗为2.1倍,峰值内存达到8GB上限。对llama.cpp进行逐节点时间测量显示,在边缘后端中路由计算占不到摩尔块计算的9%,性能差距主要源于总参数量带来的内存开销、专家调度与KV缓存压力,而非路由本身。这表明,在带宽受限的边缘硬件上,推理成本由总参数量决定,而非活跃参数量。该结论适用于此规模的单一MoE模型与两台设备,研究已发布完整测量工具与各次运行数据。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) language models are often described as ideal for resource-constrained inference. Each token activates only a small subset of experts, so the per-token compute cost, in floating-point operations (FLOPs), resembles that of a much smaller dense model. Whether that FLOP advantage survives in practice is far less clear. We ask whether MoE models actually run faster and cheaper than comparable dense models on consumer-grade and edge hardware. We benchmark OLMoE-1B-7B (1.3 B active of 6.9 B total) against three dense baselines on an Apple M2 Pro and an NVIDIA Jetson Orin Nano 8 GB through llama$.$cpp, measuring throughput, memory, and on-device energy. The answer is device-dependent: OLMoE's active-parameter advantage is only partly realised on the laptop (~10% behind the same-active Llama-3.2-1B) and erodes on the edge device (~31% behind, at 2.1$\times$ the energy per token, with peak memory at the 8 GB ceiling). Patching llama$.$cpp to time the decode graph node-by-node shows routing accounts for under 9% of MoE-block compute on the cleaner edge backend, so the gap reflects total-parameter memory footprint, expert dispatch, and KV-cache pressure rather than routing. The implication is that on bandwidth-bound edge hardware, inference cost tracks total parameters, not active ones, and sparse activation does not buy back what the device is constrained on. These findings are bounded to one MoE model at this parameter scale and two devices, and we release the full measurement harness and per-run data.

MoE边缘计算推理优化硬件评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。