通过融合CPU与内存级计算单元，提升MoE模型推理效率
TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
- 将热、温、冷专家分派至GPU/CPU/NDP最优计算单元
- 相比现有方案最高提速2.83倍，突破传统内存瓶颈
- 适合需要高吞吐量的MoE模型部署场景
为实现大容量混合专家（MoE）模型的成本效益部署，基于卸载的单GPU异构推理至关重要。当前基于GPU-CPU架构的方案受限于主机内存带宽，而新兴的GPU-NDP架构虽能通过DIMM-NDP卸载非热点专家，但非热点专家并非均一的内存密集型任务：大量‘温’专家因高GPU I/O延迟受挫，却可充分饱和NDP计算能力，暴露出关键的算力空缺。本文提出TriMoE，一种新型的GPU-CPU-NDP架构，通过协同利用支持AMX的CPU，精准将热、温、冷专家映射至最优计算单元。我们还设计了瓶颈感知的专家调度策略及预测驱动的动态重布局/负载均衡机制。实验表明，TriMoE相比最先进方案最高实现2.83倍加速。
原文摘要 · Abstract (English)
To deploy large Mixture-of-Experts (MoE) models cost-effectively, offloading-based single-GPU heterogeneous inference is crucial. While GPU-CPU architectures that offload cold experts are constrained by host memory bandwidth, emerging GPU-NDP architectures utilize DIMM-NDP to offload non-hot experts. However, non-hot experts are not a homogeneous memory-bound group: a significant subset of warm experts exists is severely penalized by high GPU I/O latency yet can saturate NDP compute throughput, exposing a critical compute gap. We present TriMoE, a novel GPU-CPU-NDP architecture that fills this gap by synergistically leveraging AMX-enabled CPU to precisely map hot, warm, and cold experts onto their optimal compute units. We further introduce a bottleneck-aware expert scheduling policy and a prediction-driven dynamic relayout/rebalancing scheme. Experiments demonstrate that TriMoE achieves up to 2.83x speedup over state-of-the-art solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。