用概率分布预测输出长度，提升大模型推理调度效率
Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
- 将输出长度建模为对数- t 分布，而非单一数值
- 新指标 TIE 在线上降低 2.31 倍每令牌延迟，线下吞吐提升 1.42 倍
- 适合追求高吞吐与低延迟的在线推理系统
为优化大模型推理调度，最短作业优先（SJF）原则通过优先处理输出较短的请求来避免队头阻塞。现有方法通常对每个请求预测一个单一输出长度以支持调度。我们认为，这种点估计无法反映大模型解码过程的随机性——输出长度本质上是不确定的，由何时采样到结束符（EOS）决定。因此，每个请求的输出长度应建模为分布而非固定值。通过对实测数据和随机解码过程的深入分析，我们发现输出长度服从重尾分布，可用对数- t 分布拟合。基于此，提出一种新指标 Tail Inflated Expectation（TIE），用于替代 SJF 中的期望输出长度，通过引入尾部概率调整期望值，以应对生成长输出的风险。在三个强基线对比下，TIE 调度器在在线推理中将每令牌延迟降低 2.31 倍，在离线数据生成中使吞吐量提升 1.42 倍。
原文摘要 · Abstract (English)
To schedule LLM inference, the \textit{shortest job first} (SJF) principle is favorable by prioritizing requests with short output lengths to avoid head-of-line (HOL) blocking. Existing methods usually predict a single output length for each request to facilitate scheduling. We argue that such a \textit{point estimate} does not match the \textit{stochastic} decoding process of LLM inference, where output length is \textit{uncertain} by nature and determined by when the end-of-sequence (EOS) token is sampled. Hence, the output length of each request should be fitted with a distribution rather than a single value. With an in-depth analysis of empirical data and the stochastic decoding process, we observe that output length follows a heavy-tailed distribution and can be fitted with the log-t distribution. On this basis, we propose a simple metric called Tail Inflated Expectation (TIE) to replace the output length in SJF scheduling, which adjusts the expectation of a log-t distribution with its tail probabilities to account for the risk that a request generates long outputs. To evaluate our TIE scheduler, we compare it with three strong baselines, and the results show that TIE reduces the per-token latency by $2.31\times$ for online inference and improves throughput by $1.42\times$ for offline data generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。