通过写时复制技术,让多LoRA代理共享大缓存,提升推理吞吐量。
ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache
- 用写时复制机制拆分共享与专属缓存,减少重复存储
- 在多种模型和任务上实现最高3.0倍的吞吐提升
- 适合需要高效部署多个专用AI代理的场景
大型语言模型的服务正转向复杂的多智能体协作流程,依赖共享上下文。尽管低秩适配(LoRA)可高效共用基础模型,但各智能体的唯一激活导致键值(KV)缓存差异,使传统前缀缓存失效,迫使系统冗余维护缓存,迅速耗尽显存并降低吞吐。为此,我们提出ForkKV,一种基于操作系统级写时复制(CoW)的新型内存管理范式。通过利用LoRA的结构特性,ForkKV将KV缓存物理分离为大规模共享部分(类比父进程内存页)和轻量级专属部分(子进程页)。为此,我们设计了DualRadixTree架构,支持新分叉智能体继承大共享缓存,并对轻量专属缓存应用CoW语义。此外,为保障高效执行,我们提出ResidualAttention专用内核,直接在片上SRAM中重建拆分后的KV缓存。在多种语言模型及实际任务数据集上的全面评估表明,ForkKV相较当前最优多LoRA服务系统,吞吐提升达3.0倍,生成质量影响微乎其微。
原文摘要 · Abstract (English)
The serving paradigm of large language models (LLMs) is rapidly shifting towards complex multi-agent workflows where specialized agents collaborate over massive shared contexts. While Low-Rank Adaptation (LoRA) enables the efficient co-hosting of these specialized agents on a single base model, it introduces a critical memory footprint bottleneck during serving. Specifically, unique LoRA activations cause Key-Value (KV) cache divergence across agents, rendering traditional prefix caching ineffective for shared contexts. This forces redundant KV cache maintenance, rapidly saturating GPU capacity and degrading throughput. To address this challenge, we introduce ForkKV, a serving system for multi-LoRA agent workflows centered around a novel memory management paradigm in OS: fork with copy-on-write (CoW). By exploiting the structural properties of LoRA, ForkKV physically decouples the KV cache into a massive shared component (analogous to the parent process's memory pages) and lightweight agent-specific components (the child process's pages). To support this mechanism, we propose a DualRadixTree architecture that allows newly forked agents to inherit the massive shared cache and apply CoW semantics for their lightweight unique cache. Furthermore, to guarantee efficient execution, we design ResidualAttention, a specialized kernel that reconstructs the disaggregated KV cache directly within on-chip SRAM. Comprehensive evaluations across diverse language models and practical datasets of different tasks demonstrate that ForkKV achieves up to 3.0x the throughput of state-of-the-art multi-LoRA serving systems with a negligible impact on generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。