TOPAS优化多智能体LLM服务中的缓存调度，减少任务完成时间。
TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
- 基于任务导向的前缀感知调度，权衡缓存保留与请求批处理
- 合成工作流上平均/尾部任务完成时间降低39.8%/49.4%
- 适用于复杂多阶段任务场景，如软件开发工作流
前缀缓存为多智能体大语言模型服务带来根本性权衡：长期保留系统提示的键值（KV）缓存可加速后续调用，但会减少用于并行请求批处理的GPU内存。在多阶段工作流中，现有调度器往往优先考虑即时前缀局部性或整体工作流进度，但在共享KV缓存预算下，单独优化任一目标可能导致下游延迟或频繁前缀替换，从而延长任务级作业完成时间（JCT）。为此，我们提出TOPAS——一种任务导向的前缀感知调度器，联合决定哪些智能体前缀应保留在缓存中，以及哪些请求应被调度执行。TOPAS通过权衡每个任务最长剩余服务路径的预期缩减与下游前缀复用的近期收益来评分候选后决策状态，同时考虑前缀迁移和抢占的成本。还引入任务级老化机制以防止饥饿。我们在SGLang框架中实现TOPAS，评估其在三个合成有向无环图（DAG）和两个MetaGPT软件开发工作流上的表现。相较于各工作负载和指标下的最优基线，TOPAS在合成工作流上将平均/99%分位数JCT分别降低最高达39.8%/49.4%，在MetaGPT-SOP上平均JCT降低9.8%，在MetaGPT-TL上平均/99%分位数JCT分别降低22.0%/26.6%。
原文摘要 · Abstract (English)
Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available for batching concurrent requests. In multi-stage workflows, existing schedulers tend to prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can prolong tasklevel job completion time (JCT) through downstream delays or frequent prefix replacement. To strike a balance, we here propose TOPAS, a Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution. TOPAS scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream prefix reuse, accounting for the costs of prefix movement and preemption. A task-level aging mechanism is also incorporated to prevent starvation. We implement TOPAS within the SGLang framework and assess its performance on three synthetic DAGs and two MetaGPT software-development workflows. Compared with the best performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%/49.4% on the synthetic workloads, while lowering mean JCT by 9.8% on MetaGPT-SOP and mean/p99 JCT by 22.0%/26.6% on MetaGPT-TL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。