通过压缩专家参数，让低内存GPU也能高效运行MoE模型。
FloE: On-the-Fly MoE Inference on Memory-constrained GPU
- 利用专家内部参数冗余，动态压缩激活专家数据。
- 在11GB显存的3090上实现8.5倍内存压缩和48.7倍加速。
- 适合资源受限设备部署大模型，性能损失仅4.4%~7.6%。
随着混合专家（Mixture-of-Experts, MoE）模型的广泛应用，对内存受限设备上的高效推理需求日益增长。尽管将专家参数卸载至CPU内存并按需加载已成为潜在解决方案，但激活专家的庞大体量仍会严重占用有限的PCIe带宽，制约其在低延迟场景下的表现。为此，我们提出FloE——一种面向内存受限GPU的实时MoE推理系统。其核心洞察在于稀疏激活专家中存在大量未被利用的冗余。FloE通过对专家内部参数矩阵采用多种压缩技术，并结合低成本稀疏预测机制，显著降低数据传输开销，在资源受限设备上实现了可感知的推理加速。实验表明，FloE在Mixtral-8x7B模型上实现每专家9.3倍的参数压缩；可在仅11GB显存的GPU上部署，内存占用减少达8.5倍；相比DeepSpeed-MII，在单张GeForce RTX 3090上实现48.7倍的推理加速，且平均性能下降仅为4.4%～7.6%。
原文摘要 · Abstract (English)
With the widespread adoption of Mixture-of-Experts (MoE) models, there is a growing demand for efficient inference on memory-constrained devices. While offloading expert parameters to CPU memory and loading activated experts on demand has emerged as a potential solution, the large size of activated experts overburdens the limited PCIe bandwidth, hindering the effectiveness in latency-sensitive scenarios. To mitigate this, we propose FloE, an on-the-fly MoE inference system on memory-constrained GPUs. FloE is built on the insight that there exists substantial untapped redundancy within sparsely activated experts. It employs various compression techniques on the expert's internal parameter matrices to reduce the data movement load, combined with low-cost sparse prediction, achieving perceptible inference acceleration in wall-clock time on resource-constrained devices. Empirically, FloE achieves a 9.3x compression of parameters per expert in Mixtral-8x7B; enables deployment on a GPU with only 11GB VRAM, reducing the memory footprint by up to 8.5x; and delivers a 48.7x inference speedup compared to DeepSpeed-MII on a single GeForce RTX 3090 - all with only a 4.4$\%$ - 7.6$\%$ average performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。