通过动态缓存优化大模型工作流，显著降低延迟并提升速度。
Accelerating Language Model Workflows with Prompt Choreography
- 用全局动态KV缓存复用消息编码，支持任意重排和并行调用。
- 单条消息延迟降低2.0至6.2倍，端到端速度提升超2.2倍。
- 适用于多智能体任务中重复计算多的场景，尤其适合高并发部署。
大型语言模型正越来越多地应用于多智能体工作流中。我们提出Prompt Choreography框架，通过维护一个动态的全局KV缓存，高效执行LLM工作流。每次LLM调用可访问先前编码消息的任意重排子集，并支持并行调用。尽管缓存消息编码有时与在新上下文中重新编码结果不同，但我们证明，在多种设置下对LLM进行微调，可使其模拟原始结果。Prompt Choreography显著降低了每条消息的延迟（时间到首个词元快2.0–6.2倍），并在某些以冗余计算为主的流程中实现了超过2.2倍的端到端加速。
原文摘要 · Abstract (English)
Large language models are increasingly deployed in multi-agent workflows. We introduce Prompt Choreography, a framework that efficiently executes LLM workflows by maintaining a dynamic, global KV cache. Each LLM call can attend to an arbitrary, reordered subset of previously encoded messages. Parallel calls are supported. Though caching messages' encodings sometimes gives different results from re-encoding them in a new context, we show in diverse settings that fine-tuning the LLM to work with the cache can help it mimic the original results. Prompt Choreography significantly reduces per-message latency (2.0--6.2$\times$ faster time-to-first-token) and achieves substantial end-to-end speedups ($>$2.2$\times$) in some workflows dominated by redundant computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。