对比边缘部署中白盒与黑盒模型压缩策略,发现蒸馏+SPTQ组合在低延迟下表现最优。
On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performance
- 采用蒸馏与SPTQ组合,实现低延迟下的高效模型压缩。
- 在移动和边缘端,含蒸馏的策略比分块策略延迟更低。
- 文本类模型适合云端部署,图像类模型更适合边缘部署。
为帮助MLOps工程师在不同部署场景中选择合适算子,本研究通过实证评估白盒(基于训练)与黑盒(非训练)算子及其组合在边缘AI环境中的精度与延迟权衡。实验涵盖3种白盒算子(QAT、剪枝、知识蒸馏)、2种黑盒算子(分块、SPTQ)及其组合(蒸馏+SPTQ、SPTQ+分块),在移动、边缘、云三类层级上对4个常用视觉与自然语言处理模型进行推理测试。结果表明:当边缘端需低延迟且可接受小至中等精度损失时,蒸馏+SPTQ(DSPTQ)组合优于非混合算子;在非混合算子中,蒸馏方案在移动与边缘端能实现更低延迟,代价为小至中等精度下降;含蒸馏的算子在资源受限的移动与边缘端延迟低于分块相关算子;对于输入数据量小的文本类模型,云端部署优于移动/边缘或其组合;而输入数据量大的图像类模型则更适合边缘部署。
原文摘要 · Abstract (English)
To help MLOps engineers decide which operator to use in which deployment scenario, this study aims to empirically assess the accuracy vs latency trade-off of white-box (training-based) and black-box operators (non-training-based) and their combinations in an Edge AI setup. We perform inference experiments including 3 white-box (i.e., QAT, Pruning, Knowledge Distillation), 2 black-box (i.e., Partition, SPTQ), and their combined operators (i.e., Distilled SPTQ, SPTQ Partition) across 3 tiers (i.e., Mobile, Edge, Cloud) on 4 commonly-used Computer Vision and Natural Language Processing models to identify the effective strategies, considering the perspective of MLOps Engineers. Our Results indicate that the combination of Distillation and SPTQ operators (i.e., DSPTQ) should be preferred over non-hybrid operators when lower latency is required in the edge at small to medium accuracy drop. Among the non-hybrid operators, the Distilled operator is a better alternative in both mobile and edge tiers for lower latency performance at the cost of small to medium accuracy loss. Moreover, the operators involving distillation show lower latency in resource-constrained tiers (Mobile, Edge) compared to the operators involving Partitioning across Mobile and Edge tiers. For textual subject models, which have low input data size requirements, the Cloud tier is a better alternative for the deployment of operators than the Mobile, Edge, or Mobile-Edge tier (the latter being used for operators involving partitioning). In contrast, for image-based subject models, which have high input data size requirements, the Edge tier is a better alternative for operators than Mobile, Edge, or their combination.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。