通过分阶段动态分配资源,实现跨服务低成本智能推理。
SERFLOW: A Cross-Service Cost Optimization Framework for SLO-Aware Dynamic ML Inference
- 将模型分成多个阶段,按请求退出比例分配资源。
- 相比传统方案,云成本降低超23%,适应突发流量。
- 适合需要弹性部署和成本控制的实时推理系统。
在函数即服务(FaaS)和基础设施即服务(IaaS)间动态调度机器学习模型分区,可在平衡处理与传输延迟的同时最小化自适应推理应用的成本。然而,现有工作常忽略实际因素,如虚拟机(VM)冷启动、长尾分布下的服务时延等。本文将每个机器学习查询建模为一个有向无环阶段序列,每个阶段由连续的稀疏模型参数块构成,并以内部或最终分类器结束,请求可在此退出。由于退出率依赖输入,单一资源配置无法适配所有查询分布。当大量请求早期退出时,基于IaaS的虚拟机易出现利用率低下;而快速扩容以应对深层请求洪流则不切实际。SERFLOW通过使用基于服务器的函数(容器)并结合阶段特异性资源分配,有效解决该问题。同时,根据请求到达情况,动态调整虚拟机与无服务器函数间的负载均衡。实验表明,该框架在保持高效响应的同时,云成本降低超过23%。
原文摘要 · Abstract (English)
Dynamic offloading of Machine Learning (ML) model partitions across different resource orchestration services, such as Function-as-a-Service (FaaS) and Infrastructure-as-a-Service (IaaS), can balance processing and transmission delays while minimizing costs of adaptive inference applications. However, prior work often overlooks real-world factors, such as Virtual Machine (VM) cold starts, requests under long-tail service time distributions, etc. To tackle these limitations, we model each ML query (request) as traversing an acyclic sequence of stages, wherein each stage constitutes a contiguous block of sparse model parameters ending in an internal or final classifier where requests may exit. Since input-dependent exit rates vary, no single resource configuration suits all query distributions. IaaS-based VMs become underutilized when many requests exit early, yet rapidly scaling to handle request bursts reaching deep layers is impractical. SERFLOW addresses this challenge by leveraging FaaS-based serverless functions (containers) and using stage-specific resource provisioning that accounts for the fraction of requests exiting at each stage. By integrating this provisioning with adaptive load balancing across VMs and serverless functions based on request ingestion, SERFLOW reduces cloud costs by over $23\%$ while efficiently adapting to dynamic workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。