arXiv:2409.17341cs.CV2024-09被引 5

通过智能跳过无变化区域,实现视频视觉的低功耗实时处理。

Energy-Efficient & Real-Time Computer Vision with Intelligent Skipping via Reconfigurable CMOS Image Sensors

  • 在传感器读出阶段动态跳过不重要的像素行或区域。
  • 在BDD100K和ImageNetVID上降低53%前端功耗,保持领先精度。
  • 适合自动驾驶与增强/虚拟现实等对能效敏感的实时场景。

当前基于视频的计算机视觉应用通常因读取并处理每帧所有像素而产生高能耗。尽管已有研究尝试通过跳过输入块或像素,并利用后端任务反馈指导跳过策略来降低能耗,但这些方法未在传感器读出阶段执行跳过,因此无法优化前端传感器能耗。此外,由于现代视觉网络延迟较长,这些方法可能不适用于实时应用。为此,本文提出一种定制的可重构CMOS图像传感器(CIS)系统,在传感器读出阶段及后续模数转换(ADC)阶段,选择性跳过帧内无事件区域或行。一种新颖的掩码算法实时引导跳过过程,同时优化前端传感器与后端神经网络。该系统可根据应用需求在标准模式与跳过模式间切换。我们在基于BDD100K和ImageNetVID的目标检测任务,以及基于OpenEDS的眼动估计任务上评估了该软硬件协同设计框架,实现了最高达53%的前端传感器能耗降低,且维持了当前最优(SOTA)精度。

原文摘要 · Abstract (English)

Current video-based computer vision (CV) applications typically suffer from high energy consumption due to reading and processing all pixels in a frame, regardless of their significance. While previous works have attempted to reduce this energy by skipping input patches or pixels and using feedback from the end task to guide the skipping algorithm, the skipping is not performed during the sensor read phase. As a result, these methods can not optimize the front-end sensor energy. Moreover, they may not be suitable for real-time applications due to the long latency of modern CV networks that are deployed in the back-end. To address this challenge, this paper presents a custom-designed reconfigurable CMOS image sensor (CIS) system that improves energy efficiency by selectively skipping uneventful regions or rows within a frame during the sensor's readout phase, and the subsequent analog-to-digital conversion (ADC) phase. A novel masking algorithm intelligently directs the skipping process in real-time, optimizing both the front-end sensor and back-end neural networks for applications including autonomous driving and augmented/virtual reality (AR/VR). Our system can also operate in standard mode without skipping, depending on application needs. We evaluate our hardware-algorithm co-design framework on object detection based on BDD100K and ImageNetVID, and gaze estimation based on OpenEDS, achieving up to 53% reduction in front-end sensor energy while maintaining state-of-the-art (SOTA) accuracy.

低功耗图像传感器实时处理跳过机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。