不依赖预测，用统计信号优化大模型推理延迟尾部表现
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
- 用轻量统计信号替代长度预测，实现无预测调度
- 在真实负载下将P99延迟降低35%-50%，TTFT降低34%-47%
- 特别适合对响应速度敏感的在线推理场景
大模型服务中序列长度变化剧烈，基于长度的调度难以实施。现有调度器依赖预测的解码长度或排序，主要关注均值指标如首次响应时间（TTFT）和总时间（TBT）。我们发现这些预测驱动策略在分布偏移、突发请求和显存压力下易失效，且即使拥有完美长度信息，对主导用户体验的尾部延迟（P90-P99）控制仍有限。本文提出一种分布感知、无需预测的调度框架，以轻量统计信号驱动软优先级提升，并协同缓存感知的抢占机制，应对混合负载下的内存耦合解码动态。在生产及开源数据集上评估显示，该方法在保持良好吞吐的同时，相比具备完美长度知识的SRPT，P99总延迟降低35%-50%，TTFT降低34%-47%，涵盖推理密集型与对话密集型任务。结果表明其为在线大模型服务中优化尾部延迟提供了稳健方案。
原文摘要 · Abstract (English)
LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。