融合事件相机与摄像头,提升复杂交通场景下的目标检测精度
Beyond conventional vision: RGB-event fusion for robust object detection in dynamic traffic scenarios
- 用事件相机弥补传统摄像头动态范围不足的缺陷
- 在恶劣光照下实现7.4%的mAP50提升,检测更鲁棒
- 适合自动驾驶、夜间行车等高动态场景应用
传统RGB摄像头在复杂交通环境(如夜间驾驶、隧道)中因动态范围受限,导致全局对比度下降,丢失纹理和边缘等高频细节,影响特征提取并降低基于帧的目标检测性能。为此,本文融合生物启发式事件相机与RGB相机,提出运动线索融合网络(MCFNet),在挑战性光照条件下实现最优时空对齐与自适应跨模态特征融合。具体地,事件校正模块(ECM)通过光流扭曲实现异步事件流与图像帧的时间对齐,并与检测网络联合优化以学习任务感知的事件表示;事件动态上采样模块(EDUM)提升事件帧的空间分辨率以匹配图像结构,确保精确时空对齐;跨模态马比融合模块(CMM)采用新型交错扫描机制实现自适应特征融合,有效整合互补信息。在DSEC-Det与PKU-DAVIS-SOD数据集上的实验表明,MCFNet在多种低光照及快速运动场景下显著优于现有方法。特别地,在DSEC-Det数据集上,mAP50提升7.4%,mAP提升1.7%。代码已公开于https://github.com/Charm11492/MCFNet。
原文摘要 · Abstract (English)
The dynamic range limitation of conventional RGB cameras reduces global contrast and causes loss of high-frequency details such as textures and edges in complex traffic environments (e.g., nighttime driving, tunnels), hindering discriminative feature extraction and degrading frame-based object detection. To address this, we integrate a bio-inspired event camera with an RGB camera to provide high dynamic range information and propose a motion cue fusion network (MCFNet), which achieves optimal spatiotemporal alignment and adaptive cross-modal feature fusion under challenging lighting. Specifically, an event correction module (ECM) temporally aligns asynchronous event streams with image frames via optical-flow-based warping, jointly optimized with the detection network to learn task-aware event representations. The event dynamic upsampling module (EDUM) enhances spatial resolution of event frames to match image structures, ensuring precise spatiotemporal alignment. The cross-modal mamba fusion module (CMM) uses adaptive feature fusion with a novel interlaced scanning mechanism, effectively integrating complementary information for robust detection. Experiments conducted on the DSEC-Det and PKU-DAVIS-SOD datasets demonstrate that MCFNet significantly outperforms existing methods in various poor lighting and fast moving traffic scenarios. Notably, on the DSEC-Det dataset, MCFNet achieves a remarkable improvement, surpassing the best existing methods by 7.4% in mAP50 and 1.7% in mAP metrics, respectively. The code is available at https://github.com/Charm11492/MCFNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。