动态调整模型精度与缓存,让大模型在高负载下仍快且准。
MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
- 运行时按需替换低影响层为量化版本,降低计算开销。
- 根据内存压力自动调节KV缓存大小,减少92.45%服务超时。
- 适合需要弹性部署的大模型应用,如实时对话系统。
在动态和突发性工作负载下高效部署大语言模型(LLMs)仍是实际应用中的关键挑战。现有服务框架和静态模型压缩技术无法适应负载波动,导致全精度服务时出现服务等级目标(SLO)违反,或静态量化下持续的精度下降。我们提出MorphServe,一种基于形态适应的动态、工作负载感知的LLM服务框架。MorphServe引入两项异步、逐标记的运行时机制:量化层交换,在高负载期选择性地将低影响层替换为量化版本;压力感知的KV缓存动态调整,根据内存压力实时改变缓存容量。这些机制支持状态保持切换,运行开销极小,并与现代调度和注意力技术完全兼容。在Vicuna和Llama系列模型上,使用真实工作负载进行的大量实验表明,MorphServe相比全精度服务,平均SLO违规率降低92.45%,P95 TTFT延迟提升2.2x–3.9x,且不损害生成质量。结果证明MorphServe是动态环境下大模型部署的一种实用且弹性的解决方案。
原文摘要 · Abstract (English)
Efficiently serving large language models (LLMs) under dynamic and bursty workloads remains a key challenge for real-world deployment. Existing serving frameworks and static model compression techniques fail to adapt to workload fluctuations, leading to either service-level objective (SLO) violations under full-precision serving or persistent accuracy degradation with static quantization. We present MorphServe, a dynamic, workload-aware LLM serving framework based on morphological adaptation. MorphServe introduces two asynchronous, token-level runtime mechanisms: quantized layer swapping, which selectively replaces less impactful layers with quantized alternatives during high-load periods, and pressure-aware KV cache resizing, which dynamically adjusts KV cache capacity in response to memory pressure. These mechanisms enable state-preserving transitions with minimum runtime overhead and are fully compatible with modern scheduling and attention techniques. Extensive experiments on Vicuna and Llama family models with real-world workloads demonstrate that MorphServe reduces average SLO violations by 92.45 percent and improves the P95 TTFT latency by 2.2x-3.9x compared to full-precision serving, without compromising generation quality. These results establish MorphServe as a practical and elastic solution for LLM deployment in dynamic environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。