让分割模型自己生成提示并融合深度信息,提升边界识别能力。
SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation
- 用自动生成的语义与空间提示替代人工输入
- 在12个数据集上均超越现有最佳模型
- 适合需要精准分割边界的视觉任务
近期,分割一切模型(SAM)在多种实例分割任务中展现出强大泛化能力,但其性能严重依赖人工提示质量。此外,常规方法使用的RGB图像缺乏深度信息,导致对空间结构感知和边界划分能力受限。为此,我们提出自提示深度感知的SAM(SPDA-SAM)。设计了语义-空间自提示模块(SSSPM),分别从SAM的图像编码器和掩码解码器中提取语义与空间提示。引入粗到细的RGB-D融合模块(C2FFM),将单目RGB图像与估计深度图特征融合:深度图中的结构信息提供粗粒度引导,深度局部变化则用于精细特征融合。据我们所知,这是首次将SAM以自提示与深度感知方式应用于实例分割。实验表明,SPDA-SAM在12个不同数据集上均优于现有先进方法,优异表现归因于自提示引导及粗细融合对空间信息损失的补偿。
原文摘要 · Abstract (English)
Recently, Segment Anything Model (SAM) has demonstrated strong generalizability in various instance segmentation tasks. However, its performance is severely dependent on the quality of manual prompts. In addition, the RGB images that instance segmentation methods normally use inherently lack depth information. As a result, the ability of these methods to perceive spatial structures and delineate object boundaries is hindered. To address these challenges, we propose a Self-prompted Depth-Aware SAM (SPDA-SAM) for instance segmentation. Specifically, we design a Semantic-Spatial Self-prompt Module (SSSPM) which extracts the semantic and spatial prompts from the image encoder and the mask decoder of SAM, respectively. Furthermore, we introduce a Coarse-to-Fine RGB-D Fusion Module (C2FFM), in which the features extracted from a monocular RGB image and the depth map estimated from it are fused. In particular, the structural information in the depth map is used to provide coarse-grained guidance to feature fusion, while local variations in depth are encoded in order to fuse fine-grained feature representations. To our knowledge, SAM has not been explored in such self-prompted and depth-aware manners. Experimental results demonstrate that our SPDA-SAM outperforms its state-of-the-art counterparts across twelve different data sets. These promising results should be due to the guidance of the self-prompts and the compensation for the spatial information loss by the coarse-to-fine RGB-D fusion operation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。