无需训练,用SAM 3实现遥感图像开放词汇语义分割
SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images
- 融合SAM 3的语义头与实例头输出,提升地物覆盖精度
- 利用存在得分过滤场景外类别,减少海量词汇带来的误检
- 首次将SAM 3拓展至遥感变化检测与3D分割任务
现有免训练开放词汇语义分割方法多基于CLIP,但在遥感场景中常因密集小目标导致定位不准或流程复杂。本文首次系统探索将可提示的分割一切模型3(SAM 3)应用于遥感开放词汇任务(2D语义分割、变化检测、3D语义分割),无需任何训练。提出一种掩码融合策略,结合SAM 3的语义分割头与Transformer解码器(实例头)输出,增强地物覆盖能力;利用存在头的置信度分数过滤场景中不存在的类别,缓解地理空间场景中大词汇量与块级处理带来的误检问题。进一步构建联合实例与像素级验证策略,实现开放词汇变化检测。在20个语义分割数据集、3个变化检测数据集及1个3D分割数据集上验证,结果表明该方法性能优异,充分展现SAM 3在遥感开放词汇任务中的潜力。代码已开源。
原文摘要 · Abstract (English)
Most existing methods for training-free open-vocabulary semantic segmentation are based on CLIP. While these approaches have made progress, they often face challenges in precise localization or require complex pipelines to combine separate modules, especially in remote sensing scenarios where numerous dense and small targets are present. Recently, Segment Anything Model 3 (SAM 3) was proposed, unifying segmentation and recognition in a promptable framework. In this paper, we present a comprehensive exploration of applying SAM 3 to the remote sensing open-vocabulary tasks (i.e., 2D semantic segmentation, change detection, and 3D semantic segmentation) without any training. First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder (instance head). This allows us to leverage the strengths of both heads for better land coverage. Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene, reducing false positives caused by the vast vocabulary sizes and patch-level processing in geospatial scenes. Furthermore, we extend our method to open-vocabulary change detection by a joint instance- and pixel-level verification strategy built directly upon our fused logits. We evaluate our method on extensive remote sensing datasets and tasks, including 20 segmentation datasets, 3 change detection datasets, and a 3D segmentation dataset. Experiments show that our method achieves promising performance, demonstrating the potential of SAM 3 for remote sensing open-vocabulary tasks. Our code is released at https://github.com/earth-insights/SegEarth-OV-3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。