动态量化MoE模型权重，显著节省显存并提升推理速度。
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
- 运行时动态调整专家权重精度，平衡内存与性能。
- 显存节省最高达72.0%，吞吐提升1.94倍，精度接近FP16。
- 适合高并发、内存受限的MoE大模型服务场景。
混合专家（MoE）是主流的大语言模型架构，兼具高效与高精度。但在依赖键值缓存（KV-cache）的服务场景中，模型权重的显存需求与不断增长的KV缓存之间存在矛盾。本文提出PagedWeight，一种新的MoE大模型服务管理方法，可在运行时动态量化模型权重，有效权衡专家权重精度与KV缓存大小。该方法充分应对任务准确性、内存消耗、吞吐/延迟之间的复杂权衡。在多个内存敏感的MoE服务场景中，相较于现有量化基线，PagedWeight显著优化了质量-内存关系：在保持FP16等效精度的前提下，最高实现72.0%的GPU显存节省和1.94×的吞吐提升；在相近内存预算下，相比其他量化方法精度最高提升39.3%，吞吐损失不超过4.1%。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We propose PagedWeight, a novel management method for MoE LLM serving that dynamically quantizes MoE model's weights at runtime and balances expert-weight precision with the KV cache sizes. PagedWeight exposes and effectively navigates the complex tradeoff between the model's task accuracy, memory consumption, and throughput/latency. Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94$\times$ throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。