arXiv:2606.29136cs.CVcs.AI2026-06

融合事件相机与图像数据,提升目标检测精度与稳定性。

CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

论文配图:CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection
图 1 · 摘自论文原文
  • 分层设计信息交互机制,逐步融合低、中、高层特征。
  • 在DSEC和PKU-DAVIS-SOD上超越单模与多模基线模型。
  • 适合做事件相机与视觉融合的算法研究者参考。

事件相机以高时间分辨率和高动态范围捕捉稀疏亮度变化,弥补传统RGB帧的不足。然而,以往多模态融合方法难以处理RGB帧与事件流之间的固有异质性,常导致噪声放大或特征冗余。本文提出跨模态信息交互变换器CMTFormer,通过分层融合机制实现高效稳定的多模态协作。首先,在浅层设计浅层对齐模块(SAM),有效融合低级特征并抑制属性差异与噪声;中层引入跨模态增强模块(CEM),利用纹理与边缘信息生成相互增强的中级特征;深层采用可学习深度融合模块(LDFM),通过可学习权重聚合高级语义,实现自适应融合。此外,设计空间先验模块以利用全局空间信息提升定位精度。在DSEC-Detection和PKU-DAVIS-SOD两个主流事件感知目标检测基准上进行大量实验,结果表明CMTFormer在单模与多模设置下均持续优于现有方法,充分验证了该范式的有效性。代码将在发表后公开。

原文摘要 · Abstract (English)

Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames. However, previous multi-modal fusion techniques typically fail to handle the inherent heterogeneity between RGB frames and event streams, thus easily leading to noise amplification or redundant feature integration during cross-modal fusion. In this paper, we propose a Cross-Modal information inTeraction transFormer, coined as CMTFormer, which hierarchically integrates RGB and event information to achieve efficient and stable multimodal collaboration. Specifically, we design a shallow-to-deep information interaction scheme. In the shallow stage, we present the Shallow Alignment Module (SAM) to achieve an efficient fusion of RGB and event low-level features, which mitigates attribute disparities and prevents noisy information. In the middle stage, we devise the Cross-modal Enhancement Module (CEM) that utilizes texture and edge information to produce mutually reinforced middle-level features. In the deep stage, we present the Learnable Deep Fusion Module (LDFM) which performs high-level information aggregation through learnable weights, thus enabling the network to adaptively fuse RGB and event clues. A Spatial Prior Module is further designed to utilize global spatial information to enhance localization accuracy. Extensive experiments are conducted on two prevalent event-based object detection benchmarks, i.e., DSEC-Detection and PKU-DAVIS-SOD. Our CMTFormer consistently surpasses the detection counterparts in both uni-modal and multi-modal settings, strongly demonstrating the effectiveness of our paradigm. Codes will be available upon publication.

多模态事件相机目标检测融合网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。