通过早退出机制实现移动端到云端的智能推理分流，降低延迟与开销。
Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach
- 基于早退出策略动态分配不同复杂度样本到移动、边缘或云处理
- 在GLUE数据集上推理成本降低超43%，准确率损失小于0.3%
- 适合资源受限场景下需低延迟高效率推理的NLP应用
深度神经网络（DNN）在多个领域表现卓越，但其庞大模型难以部署于移动、边缘及物联网等资源受限设备。为此，可采用分布式推理架构：将小型DNN（前几层）部署于移动设备，中型版本置于边缘节点，完整模型部署于云端。复杂度低（简单）的样本在移动端完成推理，中等复杂度（中等）的在边缘处理，高复杂度（困难）的则交由云端。由于样本复杂度事先未知，如何判断其应由多少层网络处理成为关键问题。本文提出DIMEE方法，利用早退出（Early Exit, EE）策略优化推理路径。该方法综合考虑推理精度与移动到边缘/云端的传输开销。在涵盖多种自然语言处理任务的GLUE数据集上的实验表明，相比全部推理在云端进行的情况，本方法可显著降低推理成本（超过43%），同时准确率下降极小（低于0.3%）。
原文摘要 · Abstract (English)
Recent advances in Deep Neural Networks (DNNs) have demonstrated outstanding performance across various domains. However, their large size is a challenge for deployment on resource-constrained devices such as mobile, edge, and IoT platforms. To overcome this, a distributed inference setup can be used where a small-sized DNN (initial few layers) can be deployed on mobile, a bigger version on the edge, and the full-fledged, on the cloud. A sample that has low complexity (easy) could be then inferred on mobile, that has moderate complexity (medium) on edge, and higher complexity (hard) on the cloud. As the complexity of each sample is not known beforehand, the following question arises in distributed inference: how to decide complexity so that it is processed by enough layers of DNNs. We develop a novel approach named DIMEE that utilizes Early Exit (EE) strategies developed to minimize inference latency in DNNs. DIMEE aims to improve the accuracy, taking into account the offloading cost from mobile to edge/cloud. Experimental validation on GLUE datasets, encompassing various NLP tasks, shows that our method significantly reduces the inference cost (> 43%) while maintaining a minimal drop in accuracy (< 0.3%) compared to the case where all the inference is made in cloud.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。