优化大模型服务调度,动态调整预填充和生成长度以降低延迟。
LLM Serving Optimization with Variable Prefill and Decode Lengths
- 基于F度量构建混合批处理策略,平衡批大小与后续解码成本。
- 在真实负载下显著降低端到端延迟,接近理论最优解。
- 适合高并发大模型服务场景,尤其适用于长短请求混杂的系统。
我们研究在固定KV缓存内存预算下,针对具有异构提示(prefill)和响应(decode)长度的大语言模型(LLM)服务的离线调度问题。提示令牌决定初始KV缓存使用,而每个生成的令牌进一步增加内存消耗,导致自回归解码过程中出现动态内存约束。给定一次性到达的n个请求,目标是随时间形成混合预填充和解码批次以最小化总端到端延迟。我们发现异构提示长度从根本上改变了调度问题:该问题为NP-hard,标准策略如先来先服务、最短输出优先和总大小优先的近似比可能无界。我们提出Sorted-F调度算法,通过一种平衡批大小与下游解码成本的F度量反复构造可行批次。证明Sorted-F在离线/积压模型中具有常数因子近似保证。我们还开发了实用实现,包括小规模实例的精确动态规划,以及适用于大规模实例的可扩展局部搜索和贪心启发式方法,还有基于LP引导和滚动时域的变体。在结合短对话与长文档摘要的公开工作负载上实验表明,基于F度量的调度始终优于标准基线,且在可处理实例中接近线性规划松弛的下界。
原文摘要 · Abstract (English)
We study offline scheduling for large language model (LLM) serving under a fixed KV-cache memory budget, where requests have heterogeneous prompt (prefill) and response (decode) lengths. Prompt tokens determine initial KV-cache usage, while each generated token further increases memory consumption, creating dynamic memory constraints during autoregressive decoding. Given a backlog of n requests arriving together, the goal is to form mixed prefill and decode batches over time to minimize total end-to-end latency. We show that heterogeneous prompt lengths fundamentally change the scheduling problem: the problem is NP-hard, and standard policies such as first-come-first-served, shortest-output-first, and total-size-based prioritization can have unbounded approximation ratios. We propose Sorted-F, a scheduling algorithm that repeatedly forms feasible batches using an F-metric that balances batch size against downstream decode cost. We prove that Sorted-F achieves a constant-factor approximation guarantee in the offline/backlogged model. We also develop practical implementations, including an exact dynamic program for small instances and scalable local-search and greedy heuristics for larger instances, as well as LP-guided and receding-horizon variants. Experiments on public workloads that combine short conversations and long-document summarization show that F-metric-based scheduling consistently reduces latency relative to standard baselines and remains close to the LP relaxation lower bound for tractable instances.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。