用Mamba模型高效融合图像与事件数据,提升复杂环境下的语义分割精度。
MambaSeg: Harnessing Mamba for Accurate and Efficient Image-Event Semantic Segmentation
- 双分支Mamba编码器并行处理图像与事件流,捕捉时空特征。
- 在DDD17和DSEC数据集上达到最优性能,计算成本显著降低。
- 适合需要实时、鲁棒多模态感知的自动驾驶与机器人场景。
语义分割是计算机视觉的基础任务,广泛应用于自动驾驶与机器人领域。尽管基于RGB的卷积神经网络和变换器方法表现优异,但在快速运动、低光照或高动态范围条件下性能下降,因帧相机存在局限。事件相机具有高时间分辨率和低延迟优势,但缺乏颜色与纹理信息,单独使用效果有限。现有融合方法多计算昂贵且侧重空间融合,忽视事件流的时间动态特性。本文提出MambaSeg,一种新型双分支语义分割框架,采用并行的Mamba编码器高效建模RGB图像与事件流。为减少跨模态歧义,引入双维度交互模块(DDIM),包含跨空间交互模块(CSIM)与跨时间交互模块(CTIM),在空间与时间维度上进行细粒度融合。该设计增强了跨模态对齐,降低歧义,充分利用两模态互补特性。在DDD17与DSEC数据集上的大量实验表明,MambaSeg实现当前最优分割性能,同时显著降低计算开销,展现其在高效、可扩展、鲁棒多模态感知中的潜力。
原文摘要 · Abstract (English)
Semantic segmentation is a fundamental task in computer vision with wide-ranging applications, including autonomous driving and robotics. While RGB-based methods have achieved strong performance with CNNs and Transformers, their effectiveness degrades under fast motion, low-light, or high dynamic range conditions due to limitations of frame cameras. Event cameras offer complementary advantages such as high temporal resolution and low latency, yet lack color and texture, making them insufficient on their own. To address this, recent research has explored multimodal fusion of RGB and event data; however, many existing approaches are computationally expensive and focus primarily on spatial fusion, neglecting the temporal dynamics inherent in event streams. In this work, we propose MambaSeg, a novel dual-branch semantic segmentation framework that employs parallel Mamba encoders to efficiently model RGB images and event streams. To reduce cross-modal ambiguity, we introduce the Dual-Dimensional Interaction Module (DDIM), comprising a Cross-Spatial Interaction Module (CSIM) and a Cross-Temporal Interaction Module (CTIM), which jointly perform fine-grained fusion along both spatial and temporal dimensions. This design improves cross-modal alignment, reduces ambiguity, and leverages the complementary properties of each modality. Extensive experiments on the DDD17 and DSEC datasets demonstrate that MambaSeg achieves state-of-the-art segmentation performance while significantly reducing computational cost, showcasing its promise for efficient, scalable, and robust multimodal perception.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。