用多尺寸模型按置信度分级推理，大幅降低大模型服务能耗。
HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing
- 根据置信度动态选择小或大模型推理，优先用节能的小模型。
- 实测能耗降低19.8倍，精度与仅用大模型相当。
- 适合需要高能效、低延迟的大规模云端AI服务部署。
巨型深度神经网络（DNN）已成为支持大规模云端AI服务准确性和鲁棒性的关键。然而，由于需部署庞大的GPU集群来存储模型分片和副本，巨型DNN的推理能耗极高，甚至超过训练阶段。现有方法只能在能效或推理精度之间权衡，无法兼顾。为此，我们提出HybridServe，一种新型混合式DNN模型服务系统，通过并行使用从小型到巨型的不同尺寸模型实现协同服务。该系统采用基于置信度的混合推理流水线，只要精度不下降，就优先使用能效更高的小型模型，从而减少对巨型DNN副本的需求。HybridServe还配备数据流规划器，用于高效划分和复制候选模型以最大化系统吞吐量。原型系统实验表明，相比当前最优的DNN服务系统，HybridServe可将能耗降低最多达19.8倍，同时保持与纯巨型DNN服务相当的精度。
原文摘要 · Abstract (English)
Giant Deep Neural Networks (DNNs), have become indispensable for accurate and robust support of large-scale cloud based AI services. However, serving giant DNNs is prohibitively expensive from an energy consumption viewpoint easily exceeding that of training, due to the enormous scale of GPU clusters needed to hold giant DNN model partitions and replicas. Existing approaches can either optimize energy efficiency or inference accuracy but not both. To overcome this status quo, we propose HybridServe, a novel hybrid DNN model serving system that leverages multiple sized versions (small to giant) of the model to be served in tandem. Through a confidence based hybrid model serving dataflow, HybridServe prefers to serve inference requests with energy-efficient smaller models so long as accuracy is not compromised, thereby reducing the number of replicas needed for giant DNNs. HybridServe also features a dataflow planner for efficient partitioning and replication of candidate models to maximize serving system throughput. Experimental results using a prototype implementation of HybridServe show that it reduces energy footprint by up to 19.8x compared to the state-of-the-art DNN model serving systems while matching the accuracy of serving solely with giant DNNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。