arXiv:2508.15919cs.DCcs.AI2025-08被引 2

HFX协同调度与弹性扩容,高效满足多任务大模型服务的严格延迟要求。

HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling

  • 通过预估请求预算并动态优先级排序,保障新旧请求均达标。
  • 相比顶尖系统,延迟降低65.82%,算力成本减少49.81%。
  • 适合需要高并发、低延迟和灵活扩展的真实场景部署。

大语言模型服务面临双重挑战:在动态多任务负载下,既要满足用户特定的服务等级目标(SLO),又要最小化计算成本。现有方法或依赖静态调度策略,或仅适用于单任务场景,难以应对真实部署中异构请求、可变提示长度和弹性扩缩容的需求。我们提出HFX,一个联合优化请求调度与模型副本弹性扩缩的生产级大模型服务系统。HFX引入一个调度器,实现对请求的主动预算估算与优先级排序,确保新请求与运行中请求均满足SLO。同时集成一个扩缩器,支持设备间(D2D)权重快速传输,显著降低冷启动延迟。系统还支持共置与解耦的prefill/decode部署模式,适应多样工作负载与云环境。在多任务负载下的大量实验表明,相比最先进系统,HFX实现了更高的SLO达成率,端到端延迟降低65.82%,NPU使用成本最多下降49.81%,性能提升达4.44倍。结果验证了面向SLO的调度与扩缩策略在实际大模型服务中的有效性,为低成本、高合规性部署提供了可靠框架。

原文摘要 · Abstract (English)

Large language model (LLM) serving faces the dual challenge of meeting strict user-specific service-level objectives (SLOs) while minimizing computational cost under dynamic, multi-task workloads. Existing approaches either rely on static scheduling policies or focus on single-task settings, limiting their applicability in real-world deployments with heterogeneous requests, variable prompt lengths, and elastic scaling requirements. We present HFX, a production LLM serving system that jointly optimizes request scheduling and elastic scaling across model replicas to satisfy diverse SLOs. HFX introduces a \textbf{scheduler} that performs proactive budget estimation and prioritization to ensure SLO compliance for both new and in-flight requests. HFX also integrates a \textbf{scaler} that supports fast device-to-device (D2D) weight transfer, reducing cold-start latency. Additionally, the system supports both colocated and disaggregated prefill/decode deployments, enabling adaptation to diverse workload patterns and cloud environments. Through extensive experiments on multi-task workloads, we demonstrate consistently higher SLO attainment, lower end-to-end latency, and lower NPU usage cost by up to 4.44$\times$, 65.82\%, and 49.81\%, respectively, compared to state-of-the-art systems. Our results highlight the effectiveness of SLO-aware scheduling and scaling in practical LLM serving, providing a robust framework for cost-efficient and SLO-compliant deployments.

大模型服务SLO优化弹性扩缩系统设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。