通过动态调整调度策略，显著降低大模型推理延迟。
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency
- 基于令牌接受率自适应调度，比单纯预测长度更准确
- 在真实场景下将平均推理延迟降低约39%
- 适合高并发大模型服务系统，尤其适用动态负载
推测解码通过使用小型推测模型（SSM）并行生成多个候选词元，并由大语言模型（LLM）验证，从而加速大语言模型（LLM）推理。然而，请求执行时间不确定，给系统调度带来挑战。现有方法仅依赖预测输出长度估算执行时间，但实际执行时间受输出长度和验证通过率共同影响。本文提出半先知调度算法LAPS-SD，根据解码过程中的请求特征动态调度。当令牌接受率动态变化时，LAPS-SD维护多个优先队列并支持跨队列抢占；一旦接受率趋于稳定，即可精确估计执行时间并合理调度。大量实验表明，相比现有最优调度方法，LAPS-SD将推理延迟降低约39%。
原文摘要 · Abstract (English)
Speculative decoding accelerates Large Language Model (LLM) inference by employing a small speculative model (SSM) to generate multiple candidate tokens and verify them using the LLM in parallel. This technique has been widely integrated into LLM inference serving systems. However, inference requests typically exhibit uncertain execution time, which poses a significant challenge of efficiently scheduling requests in these systems. Existing work estimates execution time based solely on predicted output length, which could be inaccurate because execution time depends on both output length and token acceptance rate of verification by the LLM. In this paper, we propose a semi-clairvoyant request scheduling algorithm called Least-Attained/Perceived-Service for Speculative Decoding (LAPS-SD). Given a number of inference requests, LAPS-SD can effectively minimize average inference latency by adaptively scheduling requests according to their features during decoding. When the token acceptance rate is dynamic and execution time is difficult to estimate, LAPS-SD maintains multiple priority queues and allows request execution preemption across different queues. Once the token acceptance rate becomes stable, LAPS-SD can accurately estimate the execution time and schedule requests accordingly. Extensive experiments show that LAPS-SD reduces inference latency by approximately 39\% compared to state-of-the-art scheduling methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。