融合事件相机与图像数据,提升目标检测精度与稳定性。
CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

- 分层设计信息交互机制,逐步融合低、中、高层特征。
- 在DSEC和PKU-DAVIS-SOD上超越单模与多模基线模型。
- 适合做事件相机与视觉融合的算法研究者参考。
事件相机以高时间分辨率和高动态范围捕捉稀疏亮度变化,弥补传统RGB帧的不足。然而,以往多模态融合方法难以处理RGB帧与事件流之间的固有异质性,常导致噪声放大或特征冗余。本文提出跨模态信息交互变换器CMTFormer,通过分层融合机制实现高效稳定的多模态协作。首先,在浅层设计浅层对齐模块(SAM),有效融合低级特征并抑制属性差异与噪声;中层引入跨模态增强模块(CEM),利用纹理与边缘信息生成相互增强的中级特征;深层采用可学习深度融合模块(LDFM),通过可学习权重聚合高级语义,实现自适应融合。此外,设计空间先验模块以利用全局空间信息提升定位精度。在DSEC-Detection和PKU-DAVIS-SOD两个主流事件感知目标检测基准上进行大量实验,结果表明CMTFormer在单模与多模设置下均持续优于现有方法,充分验证了该范式的有效性。代码将在发表后公开。
原文摘要 · Abstract (English)
Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames. However, previous multi-modal fusion techniques typically fail to handle the inherent heterogeneity between RGB frames and event streams, thus easily leading to noise amplification or redundant feature integration during cross-modal fusion. In this paper, we propose a Cross-Modal information inTeraction transFormer, coined as CMTFormer, which hierarchically integrates RGB and event information to achieve efficient and stable multimodal collaboration. Specifically, we design a shallow-to-deep information interaction scheme. In the shallow stage, we present the Shallow Alignment Module (SAM) to achieve an efficient fusion of RGB and event low-level features, which mitigates attribute disparities and prevents noisy information. In the middle stage, we devise the Cross-modal Enhancement Module (CEM) that utilizes texture and edge information to produce mutually reinforced middle-level features. In the deep stage, we present the Learnable Deep Fusion Module (LDFM) which performs high-level information aggregation through learnable weights, thus enabling the network to adaptively fuse RGB and event clues. A Spatial Prior Module is further designed to utilize global spatial information to enhance localization accuracy. Extensive experiments are conducted on two prevalent event-based object detection benchmarks, i.e., DSEC-Detection and PKU-DAVIS-SOD. Our CMTFormer consistently surpasses the detection counterparts in both uni-modal and multi-modal settings, strongly demonstrating the effectiveness of our paradigm. Codes will be available upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。