通过预测动态任务流中未来调用的智能缓存管理,提升大模型工作流推理效率。
Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management
- 基于历史数据和当前上下文预测未来调用的代理,指导缓存管理决策。
- 在动态工作流上实现最高1.85倍加速,静态工作流上达1.26倍性能提升。
- 适合需要高效处理复杂、可变流程的大模型应用开发者。
基于大语言模型的工作流将专用代理组合以完成复杂任务,这些代理通常共享大量上下文,使键值缓存(KV-Cache)重用成为可能。现有方法要么在代理级别管理缓存,无法利用工作流内部的重用机会;要么在工作流级别管理缓存,但假设每个工作流调用固定序列的代理。然而实际工作流往往是动态的,代理调用顺序及由此产生的缓存重用机会取决于具体任务上下文。为高效服务此类动态工作流,我们提出PBKV(预测式键值缓存管理)系统。对于每个工作流,PBKV融合历史工作流信息与目标工作流上下文,预测未来几步的代理调用。基于预测结果,估计缓存条目的重用潜力,并将高潜力条目保留在GPU内存中。为应对预测误差,PBKV在缓存淘汰与预取过程中采取保守策略。在三个工作流基准上的实验表明,与LRU相比,PBKV在动态工作流上实现最高1.85倍加速;与最先进基线KVFlow相比,在静态工作流上实现最高1.26倍加速。
原文摘要 · Abstract (English)
LLM-based workflows compose specialized agents to execute complex tasks, and these agents usually share substantial context, allowing KV-Cache reuse to save computation. Existing approaches either manage KV-Cache at agent level and fail to exploit the reuse opportunities within workflows, or manage cache at the workflow level but assume that each workflow calls a static sequence of agents. However, practical workflows are typically dynamic, where the sequence of invoked agents and thus induced cache reuse opportunities depend on the context of each task. To serve such dynamic workflows efficiently, we build a system dubbed PBKV (\textbf{P}rediction-\textbf{B}ased \textbf{KV}-Cache Management). For each workflow, PBKV predicts the agent invocations in several future steps by fusing the guidance from historical workflows and context of the target workflow. Based on the predictions, PBKV estimates the reuse potential of cache entries and keeps the high-potential entries in GPU memory. To be robust to prediction errors, PBKV utilizes the predictions conservatively during both cache eviction and prefetching. Experiments on three workflow benchmarks show that PBKV achieves up to $1.85\times$ speedup over LRU on dynamic workflows, and up to $1.26\times$ speedup over the SOTA baseline KVFlow on the static workflow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。