通过提示词预判生成长度，实现低延迟大模型服务
Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
- 基于提示词预测任务生成长度，用成对排序优化调度
- 在真实场景中将延迟降低最多15.7倍，优于默认调度器
- 兼容主流推理系统，跨模型通用无需重新训练
高效调度大语言模型（LLM）推理任务是实现低延迟和高吞吐的关键，尤其在推理长度变化大的推理型模型日益普及的背景下。传统先进先出（FCFS）策略常因首部阻塞（HOL）问题导致长任务阻塞后续短任务。本文提出PARS，一种提示感知的LLM任务调度器，通过成对排名与边界损失近似最短作业优先（SJF）调度，直接从提示词预测响应长度并决定任务顺序，实现极低开销的调度优化。PARS可无缝集成至vLLM这一领先的大模型服务系统。在多个大模型及真实推理场景（如聊天、数学、代码生成）中的大量实验表明，PARS相比vLLM默认调度器将延迟降低最多达15.7倍。跨模型评估显示，该设计具备良好泛化能力，可在不同大模型间有效调度而无需模型特定重训练。
原文摘要 · Abstract (English)
Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable. Traditional strategies like First Come, First-Serve (FCFS) often suffer from Head-of-Line (HOL) blocking, where long-running tasks delay shorter ones queued behind them. In this paper, we introduce PARS, a prompt-aware LLM task scheduler that mitigates HOL blocking by approximating shortest-job-first (SJF) scheduling through pairwise ranking with a margin ranking loss. PARS effectively predicts response-length-based task ordering directly from prompts, thereby optimizing scheduling decisions with minimal overhead. In addition, it integrates seamlessly with vLLM, a state-of-the-art LLM serving system, for the research community. Extensive experiments across multiple LLM models and real-world inference use cases, including chat, math, and code generation, demonstrate that PARS significantly reduces latency by up to 15.7x compared to the vLLM default scheduler. Cross-model evaluations demonstrate that our design generalizes effectively, allowing effective scheduling across diverse LLMs without requiring model-specific retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。