提升高优先级推理任务的准时完成率，同时降低系统干扰影响。
Strait: Perceiving Priority and Interference in ML Inference Serving
- 通过自适应模型预测数据传输与内核执行的资源竞争
- 高负载下高优任务超时率降低1.02至11.18个百分点
- 适合对延迟敏感且有优先级区分的本地部署场景
机器学习推理服务系统在多GPU环境下调度深度神经网络请求时，常因缺乏任务优先级支持和并发执行下的延迟估计不足，限制了其在本地部署中的应用。本文提出Strait系统，旨在提升高负载下双优先级推理任务的截止时间满足率。为改善延迟估计，Strait建模了数据传输过程中的潜在竞争，并通过自适应预测模型考虑内核执行的干扰。基于这些预测，系统实现优先级感知调度，提供差异化处理。在高强度负载下的评估表明，Strait使高优先级任务的截止时间违反率降低1.02至11.18个百分点，同时对低优先级任务的影响可接受。相比软件定义抢占方案，Strait还展现出更公平的性能表现。
原文摘要 · Abstract (English)
Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may restrict their applicability in on-premises scenarios. We present \emph{Strait}, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfer and accounts for kernel execution interference through an adaptive prediction model. By drawing on these predictions, it performs priority-aware scheduling to deliver differentiated handling. Evaluation results under intense workloads suggest that Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points while incurring acceptable costs on low-priority tasks. Compared to software-defined preemption approaches, Strait also exhibits more equitable performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。