arXiv:2609.07883cs.CL2026-09

动态调整提示词分块大小,让大模型推理更快更准时。

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

  • 根据请求截止时间动态选择最大可按时完成的分块大小。
  • 在25毫秒延迟目标下,吞吐量比固定分块提升39%~38%。
  • 无需调参,适合需要低延迟的实时对话和长文本场景。

连续批处理虽能提升大语言模型服务吞吐量,但长提示词预填充会延迟解码迭代,违反单令牌输出的延迟目标。分块预填充可缓解此干扰,但分块大小通常固定:小块保护解码延迟,却重复支付启动开销;大块提高预填充效率,但引发延迟峰值。本文提出SLOWeave,一种在线调度方法,选择预计能在最早活跃解码截止时间前完成的最大预填充分块。该决策无需针对工作负载调优,通过单调迭代成本模型的对数时间搜索实现。我们证明:只要解码单次迭代可行且成本预测准确,SLOWeave在保证所有活跃请求下一令牌截止时间的前提下,最大化即时预填充进度。我们在可复现的事件驱动模拟器与迭代级GPU运行时中评估该方法,覆盖对话、混合上下文、长上下文及突发性工作负载。在每输出令牌25毫秒的目标下,相较于最强固定分块基线,SLOWeave在混合请求上提升吞吐量39%,在长上下文请求上提升38%;在更严格的10毫秒目标下,增益分别达3.3倍和2.4倍。结果表明自适应分块是有效的服务原语,并提供可直接集成到迭代级LLM运行时的控制器。

原文摘要 · Abstract (English)

Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific chunk-size tuning and is computed by a logarithmic-time search over a monotone iteration-cost model. We prove that, whenever a decode-only iteration is feasible and the cost predictor is accurate, SLOWeave maximizes immediate prefill progress among decisions that preserve every active request's next-token deadline. We evaluate the method in a reproducible event-driven simulator and an iteration-level GPU runtime across chat, mixed-context, long-context, and bursty workloads. Under a 25ms time-per-output-token objective, SLOWeave improves goodput over the strongest fixed-chunk baseline by 39% on mixed requests and 38% on long-context requests. Under a stricter 10ms objective, the gains rise to 3.3$\times$ and 2.4$\times$, respectively. These results isolate adaptive chunk sizing as a useful serving primitive and provide an implementation-ready controller for integration with iteration-level LLM runtimes.

大模型推理延迟优化动态调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。