arXiv:2606.22013cs.LGcs.PF2026-06中稿 · ECSA 2026

用自适应测试框架精准评估模型服务最大承载力,避免资源浪费和系统崩溃。

Load Testing for Machine Learning Model Serving Systems at Scale

  • 通过实时性能反馈动态调整负载,找到在服务承诺下可稳定支撑的最大吞吐量。
  • 使用真实流量校准后,容量估算误差从30%降至2%~6%,提升显著。
  • 适合运维、研发及平台团队,助力构建高效可靠的机器学习服务系统。

机器学习(ML)模型服务已成为GPU资源的主要消耗者,但其容量规划仍以经验为主。资源不足会导致服务等级目标(SLO)违规和生产事故,过度配置则造成严重资源浪费。本文提出\sys,一个工业级的机器学习服务负载测试框架,采用自适应反馈搜索策略,系统性估计服务容量。该方法利用实时性能信号,结合阻尼机制、脉冲容忍和收敛检测,高效识别在SLO约束下的最大可持续吞吐量。我们通过对14个工业案例进行纵向分析,覆盖推荐、排序、视觉和自然语言处理四类模型架构,验证了系统化负载测试能显著提升GPU资源利用率与运行可靠性。此前大量模型部署存在资源不足问题,频繁引发事故;部署\sys后此类问题大幅减少。结果表明,针对ML特性的设计至关重要:使用真实流量记录进行工作负载校准,可使估算误差从约30%降至2%~6%;合理处理预热阶段可提高22.2%的准确性。进一步分析揭示模型规模与共部署效应是影响预测误差的关键因素。本文提炼出六条经验教训,并制定出机器学习负载测试的架构指南,为构建可靠高效的模型服务系统提供可操作建议。

原文摘要 · Abstract (English)

Machine learning (ML) model serving has become a dominant consumer of GPU infrastructure, yet capacity planning in these systems remains largely ad hoc. Under-provisioning leads to service-level objective (SLO) violations and production incidents, while over-provisioning results in substantial resource waste. This paper presents \sys, an industrial load testing framework for ML serving systems that systematically estimates serving capacity through an adaptive, feedback-driven search strategy. The approach leverages real-time performance signals, incorporating dampening, spike tolerance, and convergence detection to efficiently identify maximum sustainable throughput under SLO constraints. We evaluate \sys through a longitudinal analysis of 14 industrial case studies spanning four ML architecture classes: recommendation, ranking, vision, and NLP. This study demonstrates that systematic load testing leads to substantial improvements in GPU resource efficiency and operational reliability. Prior to adopting \sys, a significant fraction of model launches were under-provisioned, resulting in recurring incidents; these issues were substantially reduced after deployment. Our results show that ML-specific design decisions are critical to accurate capacity estimation: workload calibration using recorded traffic reduces estimation error from approximately 30\% to 2--6\%, while proper warmup handling yields a 22.2\% improvement in accuracy. Further analysis reveals key factors influencing prediction error, including model size and co-location effects. This paper distills six lessons and derive architectural guidelines for ML load testing, offering actionable insights for building reliable and efficient ML serving systems.

模型服务负载测试资源效率工业实践

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。