对比五种模型服务框架，发现深度学习专用框架性能更优。
On the Cost of Model-Serving Frameworks: An Experimental Evaluation
- 在四种场景下测试五类框架的推理性能
- TensorFlow Serving 在深度学习任务中表现最佳，延迟最低
- 专用框架比通用框架延迟显著更低，适合生产部署
在机器学习中，推理阶段是将预训练模型应用于新数据以生成预测的过程。用户通过与机器学习服务交互获取洞察、推荐或行动建议。因此，模型服务策略对高效部署和管理生产环境中的模型至关重要。这些策略需确保模型可用性、可扩展性、可靠性和高性能，适用于时间序列预测、图像分类、自然语言处理等实际应用。本文评估了五种广泛应用的模型服务框架（TensorFlow Serving、TorchServe、MLServer、MLflow 和 BentoML）在四种场景（恶意软件检测、加密货币价格预测、图像分类和情感分析）下的表现。结果表明，TensorFlow Serving 在深度学习模型服务方面优于其他所有框架；同时，深度学习专用框架（TensorFlow Serving 与 TorchServe）的延迟显著低于三种通用机器学习框架（BentoML、MLflow、MLServer）。
原文摘要 · Abstract (English)
In machine learning (ML), the inference phase is the process of applying pre-trained models to new, unseen data with the objective of making predictions. During the inference phase, end-users interact with ML services to gain insights, recommendations, or actions based on the input data. For this reason, serving strategies are nowadays crucial for deploying and managing models in production environments effectively. These strategies ensure that models are available, scalable, reliable, and performant for real-world applications, such as time series forecasting, image classification, natural language processing, and so on. In this paper, we evaluate the performances of five widely-used model serving frameworks (TensorFlow Serving, TorchServe, MLServer, MLflow, and BentoML) under four different scenarios (malware detection, cryptocoin prices forecasting, image classification, and sentiment analysis). We demonstrate that TensorFlow Serving is able to outperform all the other frameworks in serving deep learning (DL) models. Moreover, we show that DL-specific frameworks (TensorFlow Serving and TorchServe) display significantly lower latencies than the three general-purpose ML frameworks (BentoML, MLFlow, and MLServer).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。