将扩散模型拓展至多模态检测,融合RGB与异质2D数据
RGBX-DiffusionDet: A Framework for Multi-Modal RGB-X Object Detection Using DiffusionDet
- 设计动态通道压缩模块,实现跨模态特征交互
- 在KITTI、M³FD等数据集上超越纯RGB基线模型
- 适合需要融合深度、偏振或红外图像的检测任务
本文提出RGBX-DiffusionDet,一种将DiffusionDet模型拓展至多模态物体检测的框架,通过自适应多模态编码器融合异质2D数据(如深度、偏振、红外)与RGB图像。为促进跨模态交互,提出动态通道缩减卷积块注意力模块(DCR-CBAM),通过动态突出显著通道特征实现子网络间信息交换;进一步设计动态多层级聚合块(DMLAB),实现自适应多尺度空间特征融合。此外,引入新型正则化损失,增强通道显著性与空间选择性,生成紧凑且判别性强的特征嵌入。在RGB-Depth(KITTI)、新标注的RGB-Polarimetric数据集及RGB-Infrared(M³FD)基准数据集上进行大量实验,结果表明该方法持续优于仅使用RGB的基线DiffusionDet。模块化架构保持原有解码复杂度,确保高效性。这些成果确立了RGBX-DiffusionDet作为灵活的多模态目标检测方案,为将多样2D感知模态集成到基于扩散的检测流水线提供了新思路。
原文摘要 · Abstract (English)
This work introduces RGBX-DiffusionDet, an object detection framework extending the DiffusionDet model to fuse the heterogeneous 2D data (X) with RGB imagery via an adaptive multimodal encoder. To enable cross-modal interaction, we design the dynamic channel reduction within a convolutional block attention module (DCR-CBAM), which facilitates cross-talk between subnetworks by dynamically highlighting salient channel features. Furthermore, the dynamic multi-level aggregation block (DMLAB) is proposed to refine spatial feature representations through adaptive multiscale fusion. Finally, novel regularization losses that enforce channel saliency and spatial selectivity are introduced, leading to compact and discriminative feature embeddings. Extensive experiments using RGB-Depth (KITTI), a novel annotated RGB-Polarimetric dataset, and RGB-Infrared (M$^3$FD) benchmark dataset were conducted. We demonstrate consistent superiority of the proposed approach over the baseline RGB-only DiffusionDet. The modular architecture maintains the original decoding complexity, ensuring efficiency. These results establish the proposed RGBX-DiffusionDet as a flexible multimodal object detection approach, providing new insights into integrating diverse 2D sensing modalities into diffusion-based detection pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。