arXiv:2503.07911cs.MMcs.AI2025-03被引 5

提出VTPSeg框架,用图文提示提升遥感图像分割精度。

Visual and Text Prompt Segmentation: A Novel Multi-Model Framework for Remote Sensing

  • 结合图文提示与多模型优势,精准定位遥感目标
  • 在5个数据集上实现更高分割精度,减少冗余掩码
  • 适合遥感图像中多目标、小尺度物体的精细分割任务

像素级分割在遥感领域至关重要,基础视觉模型如CLIP和Segment Anything Model(SAM)在零样本分割任务中已展现出显著能力。然而,遥感特定挑战依然突出:首先,无明确提示约束的SAM常生成冗余掩码,增加后处理难度;其次,主要为全局特征对齐设计的CLIP模型忽视遥感中关键的局部目标,导致多目标影像中识别不准或注意力偏移;第三,两者均未在多尺度航拍视图上预训练,易造成漏检。为此,我们提出创新的VTPSeg流水线,融合Grounding DINO、CLIP与SAM的优势,实现增强的开放词汇图像分割。其中,Grounding DINO+(GD+)生成初始候选框,CLIP Filter++(CLIP++)结合视觉与文本提示过滤无关框,确保仅保留相关目标;随后,经优化的边界框作为提示输入FastSAM,完成精确分割。VTPSeg在五个主流遥感图像分割数据集上通过实验与消融研究验证了有效性。

原文摘要 · Abstract (English)

Pixel-level segmentation is essential in remote sensing, where foundational vision models like CLIP and Segment Anything Model(SAM) have demonstrated significant capabilities in zero-shot segmentation tasks. Despite their advances, challenges specific to remote sensing remain substantial. Firstly, The SAM without clear prompt constraints, often generates redundant masks, and making post-processing more complex. Secondly, the CLIP model, mainly designed for global feature alignment in foundational models, often overlooks local objects crucial to remote sensing. This oversight leads to inaccurate recognition or misplaced focus in multi-target remote sensing imagery. Thirdly, both models have not been pre-trained on multi-scale aerial views, increasing the likelihood of detection failures. To tackle these challenges, we introduce the innovative VTPSeg pipeline, utilizing the strengths of Grounding DINO, CLIP, and SAM for enhanced open-vocabulary image segmentation. The Grounding DINO+(GD+) module generates initial candidate bounding boxes, while the CLIP Filter++(CLIP++) module uses a combination of visual and textual prompts to refine and filter out irrelevant object bounding boxes, ensuring that only pertinent objects are considered. Subsequently, these refined bounding boxes serve as specific prompts for the FastSAM model, which executes precise segmentation. Our VTPSeg is validated by experimental and ablation study results on five popular remote sensing image segmentation datasets.

遥感分割图文提示多模型融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。