arXiv:2506.17869cs.CVcs.RO2025-06被引 5

用状态空间模型融合可见光与热成像,实现高效野外语义分割。

Cross-modal State Space Modeling for Real-time RGB-thermal Wild Scene Semantic Segmentation

  • 提出跨模态2D选择性扫描模块,建立双模态间状态传递机制。
  • 在CART数据集上性能领先,参数量少、计算成本低,复杂度线性增长。
  • 适合嵌入资源受限的野外机器人系统,兼具高效与强泛化能力。

RGB与热成像数据的融合能显著提升野外环境下机器人语义分割性能。然而,基于Transformer的多源数据处理方法计算开销大,难以部署于资源受限系统。为此,本文提出CM-SSM,一种基于跨模态状态空间建模的高效语义分割架构。框架包含两个核心组件:首先,设计跨模态2D选择性扫描(CM-SS2D)模块,在可见光与热成像间建立状态空间关联,通过构建跨模态视觉序列,从一模态推导另一模态的隐状态表示;其次,提出跨模态状态空间关联(CM-SSA)模块,将CM-SS2D提取的全局关联信息与卷积网络提取的局部空间特征有效融合。相比Transformer方法,CM-SSM在图像分辨率上的计算复杂度呈线性增长。实验表明,该方法在CART数据集上达到当前最优性能,参数更少、计算成本更低;在PST900数据集上的测试验证了其良好泛化能力。代码已开源。

原文摘要 · Abstract (English)

The integration of RGB and thermal data can significantly improve semantic segmentation performance in wild environments for field robots. Nevertheless, multi-source data processing (e.g. Transformer-based approaches) imposes significant computational overhead, presenting challenges for resource-constrained systems. To resolve this critical limitation, we introduced CM-SSM, an efficient RGB-thermal semantic segmentation architecture leveraging a cross-modal state space modeling (SSM) approach. Our framework comprises two key components. First, we introduced a cross-modal 2D-selective-scan (CM-SS2D) module to establish SSM between RGB and thermal modalities, which constructs cross-modal visual sequences and derives hidden state representations of one modality from the other. Second, we developed a cross-modal state space association (CM-SSA) module that effectively integrates global associations from CM-SS2D with local spatial features extracted through convolutional operations. In contrast with Transformer-based approaches, CM-SSM achieves linear computational complexity with respect to image resolution. Experimental results show that CM-SSM achieves state-of-the-art performance on the CART dataset with fewer parameters and lower computational cost. Further experiments on the PST900 dataset demonstrate its generalizability. Codes are available at https://github.com/xiaodonguo/CMSSM.

语义分割跨模态状态空间野外机器人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。