用多模态数据增强SAM,让分割更抗光照和遮挡。
Multimodal SAM-adapter for Semantic Segmentation
- 在SAM基础上加适配器,融合RGB与激光雷达/红外等多模态特征。
- 在DeLiVER、FMB、MUSES三个数据集上达到最新最好结果。
- 只在需要时引入额外模态,兼顾效率与鲁棒性,适合自动驾驶等场景。
语义分割是计算机视觉中的关键任务,在自动驾驶、医学影像和机器人等领域应用广泛。尽管深度学习推动了该领域的发展,但现有方法在弱光、遮挡和恶劣天气等挑战性条件下仍表现脆弱。为此,结合激光雷达、红外等辅助传感器数据的多模态方法应运而生,提供互补信息以提升鲁棒性。本文提出MM SAM-adapter,一种扩展分割一切模型(SAM)多模态语义分割能力的新框架。该方法通过适配器网络将融合后的多模态特征注入到SAM丰富的RGB特征中,使模型在保持强泛化能力的同时,仅在辅助模态提供有效线索时才选择性引入。实验在三个挑战性基准——DeLiVER、FMB和MUSES上验证,结果表明该框架达到当前最优性能。进一步分析显示,在分别划分出的RGB易与难子集上,其均优于对比方法,证明了多模态适应对鲁棒场景理解的有效性。
原文摘要 · Abstract (English)
Semantic segmentation, a key task in computer vision with broad applications in autonomous driving, medical imaging, and robotics, has advanced substantially with deep learning. Nevertheless, current approaches remain vulnerable to challenging conditions such as poor lighting, occlusions, and adverse weather. To address these limitations, multimodal methods that integrate auxiliary sensor data (e.g., LiDAR, infrared) have recently emerged, providing complementary information that enhances robustness. In this work, we present MM SAM-adapter, a novel framework that extends the capabilities of the Segment Anything Model (SAM) for multimodal semantic segmentation. The proposed method employs an adapter network that injects fused multimodal features into SAM's rich RGB features. This design enables the model to retain the strong generalization ability of RGB features while selectively incorporating auxiliary modalities only when they contribute additional cues. As a result, MM SAM-adapter achieves a balanced and efficient use of multimodal information. We evaluate our approach on three challenging benchmarks, DeLiVER, FMB, and MUSES, where MM SAM-adapter delivers state-of-the-art performance. To further analyze modality contributions, we partition DeLiVER and FMB into RGB-easy and RGB-hard subsets. Results consistently demonstrate that our framework outperforms competing methods in both favorable and adverse conditions, highlighting the effectiveness of multimodal adaptation for robust scene understanding. The code is available at the following link: https://github.com/iacopo97/Multimodal-SAM-Adapter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。