动态优化大模型推理中的键值缓存压缩,显著降低通信延迟。
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

- 根据服务负载实时调整缓存压缩策略,避免固定配置带来的性能损失。
- 在分布式推理中实现最高9.13倍的完成时间加速和32.8倍首字响应时间降低。
- 适合大规模部署大模型的服务系统,尤其关注通信效率与低延迟的场景。
大模型广泛应用于生产环境,推动推理系统逼近极限。去中心化大模型服务(如参数与状态分离、键值状态拆分)提升了可扩展性与成本效益,但也使键值缓存成为跨网络与存储边界的显式传输负载,成为端到端瓶颈。现有键值压缩多为静态运行时配置,而实际服务上下文随工作负载组合、带宽及SLO/质量预算动态变化,固定策略常导致次优甚至增加延迟。本文提出KVServe,首个面向去中心化大模型服务的感知服务与自适应键值通信压缩框架:(1) 将键值压缩统一为模块化策略空间,引入新组件并支持跨方法重构;(2) 设计贝叶斯分析引擎,高效搜索该空间并提炼出三维帕累托候选集,将离线搜索开销降低50倍;(3) 部署感知服务的在线控制器,结合解析延迟模型与轻量级强化学习机制,在约束下选择最优策略并修正离线到在线的偏差。集成于vLLM,在多个数据集、模型、GPU与网络环境下评估,KVServe在参数-状态分离服务中实现最高9.13倍的总完成时间加速,在键值拆分服务中实现最高32.8倍的首字响应时间减少。
原文摘要 · Abstract (English)
LLMs are widely adopted in production, pushing inference systems to their limits. Disaggregated LLM serving (e.g., PD separation and KV state disaggregation) improves scalability and cost efficiency, but it also turns KV into an explicit payload crossing network and storage boundaries, making KV a dominant end-to-end bottleneck. Existing KV compression are typically static runtime configurations, despite production service context varies over time in workload mix, bandwidth, and SLO/quality budgets. As a result, a fixed choice can be suboptimal or even increase latency. We present \emph{KVServe}, the first service-aware and adaptive KV communication compression framework for disaggregated LLM serving: KVServe (1) unifies KV compression into a modular strategy space with new components and cross-method recomposition; (2) introduces Bayesian Profiling Engine that efficiently searches this space and distills a 3D Pareto candidate set, reducing $50\times$ offline search overhead; and (3) deploys a Service-Aware Online Controller that combines an analytical latency model with a lightweight bandit to select profiles under constraints and correct offline-to-online mismatch. Integrated into vLLM and evaluated across datasets, models, GPUs and networks, KVServe achieves up to $9.13\times$ JCT speedup in PD-separated serving and up to $32.8\times$ TTFT reduction in KV-disaggregated serving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。