arXiv:2505.07556cs.CV2025-05被引 2

用自监督方法在FPGA上实现毫秒级事件感知,兼顾精度与实时性。

Self-Supervised Event Representations: Towards Accurate, Real-Time Perception on SoC FPGAs

  • 用GRU网络直接编码事件时间戳和极性,避免时间离散化
  • 在Gen1和1Mpx数据集上分别提升2.4%和0.6%的mAP
  • 首个在SoC FPGA上实现的循环事件表示,功耗仅1-2瓦

事件相机相比传统帧式传感器具有微秒级时间分辨率、光照变化鲁棒性和低功耗等优势。然而,其稀疏、异步的事件流处理仍具挑战。现有方法分为两类:一类是直接用脉冲神经网络或图卷积网络处理事件数据,但常牺牲性能;另一类是通过手工聚合函数将事件转为稠密表示,虽提升准确率却损失时间保真度。本文提出一种自监督事件表征(SSER)方法,利用门控循环单元(GRU)网络实现像素级事件时间戳与极性的精确编码,无需时间离散化。递归层以自监督方式训练,最大化事件时间编码保真度。推理阶段异步生成事件表示,兼容高吞吐传感器。实验表明,SSER优于基于聚合的基线,在Gen1和1 Mpx物体检测数据集上分别提升2.4%和0.6%的mAP。此外,论文首次在系统级芯片FPGA上实现了递归事件表示,达到亚微秒级延迟,功耗为1-2瓦,适用于实时、低功耗场景。代码已开源:https://github.com/vision-agh/RecRepEvent。

原文摘要 · Abstract (English)

Event cameras offer significant advantages over traditional frame-based sensors. These include microsecond temporal resolution, robustness under varying lighting conditions and low power consumption. Nevertheless, the effective processing of their sparse, asynchronous event streams remains challenging. Existing approaches to this problem can be categorised into two distinct groups. The first group involves the direct processing of event data with neural models, such as Spiking Neural Networks or Graph Convolutional Neural Networks. However, this approach is often accompanied by a compromise in terms of qualitative performance. The second group involves the conversion of events into dense representations with handcrafted aggregation functions, which can boost accuracy at the cost of temporal fidelity. This paper introduces a novel Self-Supervised Event Representation (SSER) method leveraging Gated Recurrent Unit (GRU) networks to achieve precise per-pixel encoding of event timestamps and polarities without temporal discretisation. The recurrent layers are trained in a self-supervised manner to maximise the fidelity of event-time encoding. The inference is performed with event representations generated asynchronously, thus ensuring compatibility with high-throughput sensors. The experimental validation demonstrates that SSER outperforms aggregation-based baselines, achieving improvements of 2.4% mAP and 0.6% on the Gen1 and 1 Mpx object detection datasets. Furthermore, the paper presents the first hardware implementation of recurrent representation for event data on a System-on-Chip FPGA, achieving sub-microsecond latency and power consumption between 1-2 W, suitable for real-time, power-efficient applications. Code is available at https://github.com/vision-agh/RecRepEvent.

事件相机自监督FPGA实时感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。