让高优先级大模型任务更快恢复，提升资源调度效率55%。
Topology-aware Preemptive Scheduling for Co-located LLM Workloads
- 根据资源拓扑关系智能预占，确保释放资源匹配高优任务需求。
- 在混合负载场景下使整体调度性能提升55%。
- 适合需要低延迟响应的在线大模型服务部署场景。
通过共置（co-location）方式将不同大语言模型工作负载统一部署于共享资源池，可有效降低成本。例如，长周期聊天服务通常具有昼夜流量规律，可将批处理任务安排在连续高峰间的资源空档期，从而实现集群范围内的资源饱和利用。然而，这些异构工作负载常有不同业务优先级，可通过抢占机制实现资源弹性调度。但各任务对资源拓扑结构存在偏好，被低优先级实例释放的资源可能无法满足高优先级在线服务（通常对延迟敏感）的拓扑要求。其根本原因在于资源调度器缺乏拓扑感知能力，尤其在抢占过程中。为此，本文提出一种细粒度的拓扑感知抢占调度方法，确保被抢占任务释放的资源能以保证或尽力而为的方式满足高优先级抢占者的拓扑亲和性需求。该动态对齐机制显著提升了抢占效率，使大模型工作负载的整体调度性能提升55%。
原文摘要 · Abstract (English)
Hosting diverse large language model workloads in a unified resource pool through co-location is cost-effective. For example, long-running chat services generally follow diurnal traffic patterns, which inspire co-location of batch jobs to fulfill resource valleys between successive peaks, and thus to saturate resource allocation in cluster-wide scope. These heterogeneous workloads often have different business priorities, and therefore preemption can be leveraged for resource elasticity. However, workloads often have distinct topology preferences as well. The resources released by lower-priority instances may fail to meet the requirements of high-priority online services which are usually latency-sensitive. The root cause behind such mis-match is a lack of topology awareness of resource scheduler, especially during preemption. To bridge this gap, we develop a fine-grained topology-aware method for preemptive scheduling of hybrid workloads. The method ensures that the resources freed by preempted tasks adhere to the topological affinity needs of high-priority preemptors in a guaranteed or best-effort manner. This dynamic alignment significantly increases the efficiency of preemption and improves overall scheduled performance for LLM workloads by $55\%$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。