arXiv:2501.15119cs.CVeess.IV2025-01

直接在拜耳域用运动估计加速视频视觉任务,大幅降低计算开销。

Leveraging Motion Estimation for Efficient Bayer-Domain Computer Vision

  • 在卷积层中嵌入滑动窗口运动估计,通过预测与残差优化减少帧间冗余计算。
  • 端到端框架实现超过70%的浮点运算量减少,精度损失极小。
  • 适用于多种基于卷积的模型,首次实现运动估计直接加速原始传感器数据处理。

现有计算机视觉处理流程通过图像传感器以拜耳格式采集像素信息,随后由图像信号处理器(ISP)逐像素转换为RGB,再由视频卷积网络(VCN)逐帧处理。两者均计算复杂且功耗高、延迟大。本文提出新框架,跳过ISP,利用运动估计在拜耳域直接加速视频视觉任务。引入基于运动估计的视频卷积(MEVC),将滑动窗口运动估计融入每层卷积,实现帧间预测与残差修正,减少重复计算。该设计弥合了块级运动估计与空间卷积之间的结构鸿沟,支持原始拜耳输入,实现跨视频语义分割、深度估计和目标检测基准上超过70%的浮点运算量减少,精度损失微乎其微,使用合成拜耳转换与真实拜耳视频数据集验证。该框架可泛化至多种卷积模型,是首个有效复用运动估计加速原始传感器数据视频视觉处理的工作。

原文摘要 · Abstract (English)

Existing computer vision processing pipeline acquires visual information using an image sensor that captures pixel information in the Bayer pattern. The raw sensor data are then processed using an image signal processor (ISP) that first converts Bayer pixel data to RGB on a pixel by pixel basis, followed by video convolutional network (VCN) processing on a frame by frame basis. Both ISP and VCN are computationally expensive with high power consumption and latency. In this paper, we propose a novel framework that eliminates the ISP and leverages motion estimation to accelerate video vision tasks directly in the Bayer domain. We introduce Motion Estimation-based Video Convolution (MEVC), which integrates sliding-window motion estimation into each convolutional layer, enabling prediction and residual-based refinement that reduces redundant computations across frames. This design bridges the structural gap between block-based motion estimation and spatial convolution, enabling accurate, low-cost processing. Our end-to-end pipeline supports raw Bayer input and achieves over 70\% reduction in FLOPs with minimal accuracy degradation across video semantic segmentation, depth estimation, and object detection benchmarks, using both synthetic Bayer-converted and real Bayer video datasets. This framework generalizes across convolution-based models and marks the first effective reuse of motion estimation for accelerating video computer vision directly from raw sensor data.

拜耳域运动估计视频推理低功耗

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。