用语音长度预判处理时间，优化语音识别服务延迟。
Duration Aware Scheduling for ASR Serving Under Workload Drift
- 根据音频时长预估任务耗时，改进调度策略。
- 相比基准，中位延迟降低73%，尾部延迟增加不超过24%。
- 适合高并发语音识别系统，对延迟敏感场景有帮助。
大规模自动语音识别（ASR）服务中的调度策略直接影响端到端延迟。然而，主流服务引擎仍采用先进先出（FCFS）调度，忽视请求时长差异，在负载漂移下导致队头阻塞。我们发现语音时长是Whisper等模型任务处理时间的准确代理，据此提出基于时长感知的调度方案。将经典算法最短作业优先（SJF）和最高响应比优先（HRRN）集成至vLLM，在真实与漂移负载下评估。在LibriSpeech test-clean数据集上，相比基线，SJF在高负载下将中位端到端延迟降低高达73%，但使90百分位尾部延迟上升最多97%，因长请求出现饥饿。HRRN缓解此权衡：中位延迟降低最多28%，同时将尾部延迟恶化控制在24%以内。该效果在负载漂移下依然稳定，无吞吐损失，每请求调度开销小于0.1毫秒。
原文摘要 · Abstract (English)
Scheduling policies in large-scale Automatic Speech Recognition (ASR) serving pipelines play a key role in determining end-to-end (E2E) latency. Yet, widely used serving engines rely on first-come-first-served (FCFS) scheduling, which ignores variability in request duration and leads to head-of-line blocking under workload drift. We show that audio duration is an accurate proxy for job processing time in ASR models such as Whisper, and use this insight to enable duration-aware scheduling. We integrate two classical algorithms, Shortest Job First (SJF) and Highest Response Ratio Next (HRRN), into vLLM and evaluate them under realistic and drifted workloads. On LibriSpeech test-clean, compared to baseline, SJF reduces median E2E latency by up to $73\%$ at high load, but increases $90$th-percentile tail latency by up to $97\%$ due to starvation of long requests. HRRN addresses this trade-off: it reduces median E2E latency by up to $28\%$ while bounding tail-latency degradation to at most $24\%$. These gains persist under workload drift, with no throughput penalty and $<0.1$\,ms scheduling overhead per request.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。