对比四类边缘推理框架，实测发现不同硬件下最佳方案各异。
Benchmarking Edge Inference Strategies for Deep Learning Models in Industrial Machine Vision
- 测试四种主流边缘推理框架在工业场景下的表现
- OpenVINO在CPU上最快，TensorRT在GPU上最快
- Transformer模型下TensorRT反而不如原生PyTorch
在低延迟、数据安全或网络受限的工业机器视觉系统中，边缘部署通常是首选方案。尽管已有多种框架可优化边缘设备上的推理性能，但针对工业部署条件下的系统性比较仍较少。本文对四种广泛使用的机器视觉推理方法——原生PyTorch、ONNX Runtime、OpenVINO和TensorRT进行了对比研究。评估涵盖多种基于CPU和GPU的硬件平台，以及传统卷积神经网络与基于Transformer的视觉模型。实验结果表明，在所测试的硬件和模型条件下，OpenVINO在CPU上实现最低推理时间，而TensorRT在GPU上表现最优；然而，对于本研究所考察的Transformer模型，TensorRT并未优于原生PyTorch。
原文摘要 · Abstract (English)
Edge deployment is often the preferred solution for industrial machine vision systems when low latency, data security, or limited connectivity are critical requirements. Several frameworks are available to optimise inference on edge devices; however, relatively few studies have systematically compared their inference-time performance under industrial deployment conditions. In this work, we present a comparative study of four widely used approaches for machine vision inference in industrial settings: plain PyTorch, ONNX Runtime, OpenVINO, and TensorRT. The evaluation focuses on inference time, covers several CPU- and GPU-based hardware platforms, and includes both conventional convolutional neural networks and a transformer-based vision model. For the hardware platforms and models evaluated, the results show that OpenVINO achieves the lowest inference time on CPUs, while TensorRT achieves the lowest inference time on GPUs. However, TensorRT does not outperform plain PyTorch for the transformer-based model considered in this study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。