统一多模态图像语义分割，无需为每种传感器重做模型
SegRGB-X: General RGB-X Semantic Segmentation Model
- 用适配器微调的视觉语言模型实现跨模态理解
- 在5个不同数据集上达到65.03% mIoU，超越专用多模态方法
- 适合需要跨传感器部署分割模型的研究者
任意传感器模态下的语义分割因传感器特性差异面临巨大挑战，传统方法需为每种模态单独设计，导致开发冗余。本文提出一种通用的任意模态语义分割框架，统一多模态分割任务。方法包含三项核心创新：(1) 模态感知的CLIP（MA-CLIP），通过LoRA微调提供模态特定的场景理解引导；(2) 模态对齐嵌入，捕捉细粒度特征；(3) 领域特定优化模块（DSRM），实现动态特征调整。在包含事件、热成像、深度、偏振和光场五种互补模态的五个多样化数据集上评估，模型性能优于现有专用多模态方法，达到65.03%的mIoU，实现当前最佳表现。代码将在论文接收后公开。
原文摘要 · Abstract (English)
Semantic segmentation across arbitrary sensor modalities faces significant challenges due to diverse sensor characteristics, and the traditional configurations for this task result in redundant development efforts. We address these challenges by introducing a universal arbitrary-modal semantic segmentation framework that unifies segmentation across multiple modalities. Our approach features three key innovations: (1) the Modality-aware CLIP (MA-CLIP), which provides modality-specific scene understanding guidance through LoRA fine-tuning; (2) Modality-aligned Embeddings for capturing fine-grained features; and (3) the Domain-specific Refinement Module (DSRM) for dynamic feature adjustment. Evaluated on five diverse datasets with different complementary modalities (event, thermal, depth, polarization, and light field), our model surpasses specialized multi-modal methods and achieves state-of-the-art performance with a mIoU of 65.03%. The codes will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。