arXiv:2412.04149cs.CVcs.AI2024-12被引 8

用事件相机提升目标检测速度,解决帧与事件时间不同步问题。

Frequency-Adaptive Low-Latency Object Detection Using Events and Frames

  • 通过对齐模块融合高频事件与低频图像,解决时间错位问题。
  • 在PKU-DAVIS-SOD上相比SODFormer提升9.8点mAP,参数仅四分之一。
  • 支持事件与图像频率差异达80倍,适合实时系统部署。

将事件相机与RGB图像融合用于目标检测,可利用事件相机在恶劣环境下的鲁棒性及RGB相机提供的丰富语义信息。然而,两个关键挑战:事件的低延迟与图像的高延迟;训练时时间稀疏标签与推理时连续流之间的不匹配,严重阻碍了高频融合检测的发展。为此,我们提出频率自适应低延迟目标检测器FAOD。FAOD通过对齐模块将低频RGB帧与高频事件对齐,强化跨模态风格和空间邻近性以缓解事件-图像不匹配。我们进一步提出时间偏移训练策略,强制模型对时移后的事件-图像对及其原始表示进行预测,使其与事件对齐标注保持一致。该策略使网络以高频事件数据为主参考,低频图像为补充,保留事件流的低延迟特性,实现高频检测。此外,我们发现经校正的事件-图像对在从低训练频率到更高推理频率的泛化能力优于仅使用事件数据。在PKU-DAVIS-SOD和DSEC-Detection数据集上的大量实验表明,FAOD达到当前最优性能。具体而言,在PKU-DAVIS-SOD数据集上,全配对事件-图像数据下,mAP相较SODFormer提升9.8点,参数量仅为其四分之一;即使在事件-图像频率相差80倍的情况下,仍保持仅下降3点的稳定性能。

原文摘要 · Abstract (English)

Fusing Events and RGB images for object detection leverages the robustness of Event cameras in adverse environments and the rich semantic information provided by RGB cameras. However, two critical mismatches: low-latency Events \textit{vs.}~high-latency RGB frames; temporally sparse labels in training \textit{vs.}~continuous flow in inference, significantly hinder the high-frequency fusion-based object detection. To address these challenges, we propose the \textbf{F}requency-\textbf{A}daptive Low-Latency \textbf{O}bject \textbf{D}etector (FAOD). FAOD aligns low-frequency RGB frames with high-frequency Events through an Align Module, which reinforces cross-modal style and spatial proximity to address the Event-RGB Mismatch. We further propose a training strategy, Time Shift, which enforces the module to align the prediction from temporally shifted Event-RGB pairs and their original representation, that is, consistent with Event-aligned annotations. This strategy enables the network to use high-frequency Event data as the primary reference while treating low-frequency RGB images as supplementary information, retaining the low-latency nature of the Event stream toward high-frequency detection. Furthermore, we observe that these corrected Event-RGB pairs demonstrate better generalization from low training frequency to higher inference frequencies compared to using Event data alone. Extensive experiments on the PKU-DAVIS-SOD and DSEC-Detection datasets demonstrate that our FAOD achieves SOTA performance. Specifically, in the PKU-DAVIS-SOD Dataset, FAOD achieves 9.8 points improvement in terms of the mAP in fully paired Event-RGB data with only a quarter of the parameters compared to SODFormer, and even maintains robust performance (only a 3 points drop in mAP) under 80$\times$ Event-RGB frequency mismatch.

目标检测事件相机多模态低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。