在微控制器上实现低功耗实时目标检测，支持CNN与Transformer模型
MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes
- 交替使用高/低分辨率推理，降低计算开销
- 融合多帧置信度提升检测精度，保持mAP达49.0（CNN）和48.7（Transformer）
- 首次实现在MCU上实时运行Transformer级视频目标检测
现代智能视觉传感器需具备本地智能以处理视频流，因云端计算常受限于带宽、延迟和隐私问题。但这类传感系统通常依赖超低功耗微控制器（MCU），内存与算力有限，使得需要特征存储或多帧缓冲的传统视频目标检测方法不可行。为此，我们提出针对MCU嵌入式视觉节点的多分辨率重评分ByteTrack（MR2-ByteTrack）方法。该方法通过交替进行全分辨率与低分辨率推理降低计算成本，利用ByteTrack跨帧关联检测结果，并通过重评分算法结合多帧置信度得分修正误分类。我们在基于CNN和Transformer的检测器上验证了该方法的通用性，实验表明其在ImageNetVID上分别达到49.0（CNN）和48.7（Transformer）的mAP，同时将乘加操作减少53%（CNN）和32%（Transformer）。部署于GAP9超低功耗RISC-V多核MCU时，相比仅处理全分辨率图像，能节省高达55%能耗，首次实现MCU级嵌入式节点上的实时Transformer视频目标检测。代码见https://github.com/Bomps4/Multi_Resolution_Rescored_ByteTrack/tree/IEEE_Access
原文摘要 · Abstract (English)
Modern smart vision sensors need on-device intelligence to process video streams, as cloud computing is often impractical due to bandwidth, latency, and privacy constraints. However, these sensory systems typically rely on ultra-low-power microcontrollers (MCUs) with limited memory and compute, making conventional video object detection methods, which require feature storage or multi-frame buffering, unfeasible. To address this challenge, we introduce Multi-Resolution Rescored ByteTrack (MR2-ByteTrack), a Video Object Detection (VOD) method tailored for MCU-based embedded vision nodes. MR2-ByteTrack reduces computational cost by alternating between full- and low-resolution inference, while linking detections across frames via ByteTrack and correcting misclassifications through the Rescore algorithm, which applies probability union rules to aggregate detection confidence scores across frames. We apply our approach to both a CNN-based detector and a Transformer-based model, demonstrating its generality across architectures with fundamentally different spatial processing. Experiments on ImageNetVID demonstrate that MR2-ByteTrack maintains accuracy, achieving mAP scores of up to 49.0 for the CNN-based models and 48.7 for the Transformer, while reducing multiply-accumulate operations by as much as 53\% for the CNNs and 32\% for the Transformer. When deployed on GAP9, an ultra-low-power RISC-V multicore MCU, our method yields up to 55\% energy savings compared to processing only full-resolution images, enabling the first real-time Transformer-based VOD on an MCU-class embedded vision node. Code available at https://github.com/Bomps4/Multi_Resolution_Rescored_ByteTrack/tree/IEEE_Access
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。