arXiv:2608.15383cs.LG2026-08

让专家模型推理更省显存,还能保持高效率和高精度。

Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

论文配图:Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference
图 1 · 摘自论文原文
  • 只对被选中的专家做4位量化,用专用缓存加速执行
  • 显存占用降低87%,推理速度提升50%以上,精度几乎不变
  • 适合需要高效部署大模型的工程师和研究者

稀疏混合专家(MoE)语言模型通过每令牌仅激活少量专家来减少计算量,但部署时仍需存储和传输全部专家。我们提出ExactMoE,一种推理设计:仅对被路由的专家应用对称分组128的4位权重量化,将这些专家以原生MARLIN格式存于固定主机内存,并通过可配置的GPU驻留槽缓存与融合分组MoE核执行所有选定专家。路由器、注意力、嵌入层、归一化层和语言模型头保持在BF16。'Exact'指完整专家可用且不改变top-k路由策略:无专家被剪枝、替换或强制移至CPU。它不表示数值等同于BF16模型。在单张NVIDIA L4上评估OLMoE-1B-7B-0924-Instruct,16槽配置将峰值保留显存从14.168 GiB降至1.836 GiB(节省87.04%),同时保留81.85%的BF16解码吞吐量。全驻留64槽配置达31.923 tokens/s,对比BF16的21.662 tokens/s提升显著,仅需4.061 GiB显存。在12,450个零样本多选题中,ExactMoE获得70.3534%归一化准确率,相比BF16的70.8996%保留99.23%。在匹配的16令牌消融实验中,融合分组执行速度是顺序W4参考的1.97倍。结果揭示了完整专家MoE推理在显存-吞吐量之间的实用边界。

原文摘要 · Abstract (English)

Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank. We present ExactMoE, an inference design that applies symmetric group-128 four-bit weight quantization only to routed experts, stores those experts in kernel-native MARLIN form in pinned host memory, and executes all selected experts through a configurable GPU-resident slot cache and fused grouped MoE kernels. The router, attention, embeddings, normalization layers, and language-model head remain in BF16. "Exact" refers to complete expert availability and an unchanged top-k routing procedure: no expert is pruned, substituted, or forced to execute on the CPU. It does not imply numerical identity with the BF16 model. On OLMoE-1B-7B-0924-Instruct, evaluated on a single NVIDIA L4, a 16-slot configuration reduces peak reserved GPU memory from 14.168 to 1.836 GiB (87.04%) while retaining 81.85% of BF16 decode throughput. A fully resident 64-slot configuration reaches 31.923 tokens/s versus 21.662 tokens/s for BF16 while reserving 4.061 GiB. Across 12,450 zero-shot multiple-choice questions, ExactMoE obtains 70.3534% normalized accuracy versus 70.8996% for BF16, retaining 99.23% of the baseline accuracy. In a matched 16-token ablation, fused grouped execution is 1.97x as fast as a sequential W4 reference. These results identify a practical memory-transfer-throughput frontier for complete-expert MoE inference.

MoE4位量化显存优化推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。