将CNN推理分拆在DPU和GPU上，显著降低边缘设备延迟。
DPU or GPU for Accelerating Neural Networks Inference -- Why not both? Split CNN Inference
- DPU处理输入图像的前几层，GPU处理剩余层，实现异步流水线加速。
- 相比纯DPU或纯GPU，延迟最高降低3.37倍，模型精度损失极小。
- 用图神经网络自动预测最佳分层位置，准确率达96.27%，适合边缘部署。
边缘设备上的视频与图像流需低延迟处理。为应对这一挑战，神经网络被广泛应用，现有工作多聚焦于使用单一硬件（如GPU、FPGA、DPU）加速。本文提出在Versal VCK190的DPU与NVIDIA RTX 2080 GPU之间分割CNN推理（Split CNN Inference）。DPU执行初始卷积层，靠近数据源处理；后续层由GPU异步流水线执行，减少数据传输开销。同时提出基于图神经网络（GNN）的分层预测方法，自动确定最优分割点。对LeNet-5、ResNet18/50/101/152、VGG16、MobileNetv2等模型测试显示，相比纯DPU执行，延迟最高降低2.48倍；相比纯GPU执行，最高降低3.37倍。训练后的GNN模型分层预测准确率达96.27%。
原文摘要 · Abstract (English)
Video and image streaming on edge devices requires low latency. To address this, Neural Networks (NNs) are widely used, and prior work mainly focuses on accelerating them with single hardware units such as Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), and Deep Learning Processing Units (DPUs). However, further reductions in latency can be observed by combining these units. In this paper, partitioning CNN inference across DPU and GPU (Split CNN Inference) is proposed. The first partition runs on the AI engines (DPU) of a Versal VCK190, which consists of initial CNN layers processing the input images. The DPU processes the first partition near the source of the data. Pipelined asynchronously, a GPU runs the remaining layers. The GPU (NVIDIA RTX 2080) processes the second partition, albeit having reduced the data transfer between the data source (storage/camera) and the GPU. Furthermore, a Graph Neural Network (GNN)-based partition index prediction method is proposed to automate the partitioning of CNNs needed for Split Inference. Well established models such as LeNet-5, ResNet18/50/101/152, VGG16, and MobileNetv2 are analyzed. Results demonstrate up to 2.48x latency improvement over DPU-only execution and up to 3.37x over GPU-only execution. The trained GNN model splits the layers between the appropriate devices with 96.27% accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。