轻量模型LMIINet在FPGA上实现90%准确率实时语义分割。
Real-Time Semantic Segmentation on FPGA for Autonomous Vehicles Using LMIINet with the CGRA4ML Framework
- 用轻量LMIINet+CGRA4ML框架适配FPGA硬件约束
- 20帧/秒下达90%像素准确率，mIoU 45%，延迟50.1毫秒
- 适合自动驾驶场景，比GPU更省电且精度可比
语义分割是计算机视觉的基础问题，在自动驾驶等实时应用中尤为重要。本文提出基于FPGA的实时语义分割实现，采用轻量级LMIINet架构与面向机器学习的粗粒度可重构阵列（CGRA4ML）硬件框架。模型在Cityscapes数据集上通过8位量化感知训练（QAT），内存占用减少四倍，支持高效定点计算。为适配CGRA4ML限制，简化了跳跃连接，采用深度可分离卷积、1A-1卷积等硬件友好操作，并重构了部分扁平化变换模块。系统在ZCU104 FPGA板上实现约90%像素准确率和45%平均交并比（mIoU），运行速度达20帧/秒（50.1毫秒延迟）。结果表明，CGRA4ML通过灵活映射现代网络层及利用外部存储器处理跳跃连接，为实现高性能实时语义分割提供了路径，在能效上优于传统GPU，同时保持竞争力精度。代码已公开于https://github.com/STAmirr/cgra4ml_semantic_segmentation。
原文摘要 · Abstract (English)
Semantic segmentation has emerged as a fundamental problem in computer vision, gaining particular importance in real-time applications such as autonomous driving. The main challenge is achieving high accuracy while operating under computational and hardware constraints. In this research, we present an FPGA-based implementation of real-time semantic segmentation leveraging the lightweight LMIINet architecture and the Coarse-Grained Reconfigurable Array for Machine Learning (CGRA4ML) hardware framework. The model was trained using Quantization-Aware Training (QAT) with 8-bit precision on the Cityscapes dataset, reducing memory footprint by a factor of four while enabling efficient fixed-point computations. Necessary modifications were applied to adapt the model to CGRA4ML constraints, including simplifying skip connections, employing hardware-friendly operations such as depthwise-separable and 1A-1 convolutions, and redesigning parts of the Flatten Transformer. Our implementation achieves approximately 90% pixel accuracy and 45% mean Intersection-over-Union (mIoU), operating in real-time at 20 frames per second (FPS) with 50.1 ms latency on the ZCU104 FPGA board. The results demonstrate the potential of CGRA4ML, with its flexibility in mapping modern layers and off-chip memory utilization for skip connections, provides a path for implementing advanced semantic segmentation networks on FPGA for real-time applications to outperform traditional GPU solutions in terms of power efficiency while maintaining competitive accuracy. The code for this project is publicly available at https://github.com/STAmirr/ cgra4ml_semantic_segmentation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。