用基础模型提升遥感图像指代分割精度，实现文本与视觉精准对齐。
RSRefSeg: Referring Remote Sensing Image Segmentation with Foundation Models
- 融合CLIP的全局与局部语义，生成指向性视觉激活特征。
- 在RRSIS-D数据集上显著超越现有方法，达新基准水平。
- 适合遥感目标提取、智能地图解析等应用场景。
指代遥感图像分割对于通过自由格式文本输入实现细粒度视觉理解至关重要，可增强遥感应用中的场景与目标提取能力。现有研究主要利用预训练语言模型编码文本描述，并与视觉模态对齐，以表达相关视觉特征，但常难以建立细粒度语义概念间的稳健对齐，导致文本与视觉信息表示不一致。为此，我们提出一个遥感图像指代分割基础模型RSRefSeg。RSRefSeg采用CLIP进行视觉与文本编码，利用全局和局部文本语义作为滤波器，在潜在空间中生成指代相关的视觉激活特征。这些激活特征作为输入提示，驱动SAM通过其强大的视觉泛化能力优化分割掩码。在RRSIS-D数据集上的实验结果表明，RSRefSeg优于现有方法，验证了基础模型在增强多模态任务理解方面的有效性。代码已开源： https://github.com/KyanChen/RSRefSeg。
原文摘要 · Abstract (English)
Referring remote sensing image segmentation is crucial for achieving fine-grained visual understanding through free-format textual input, enabling enhanced scene and object extraction in remote sensing applications. Current research primarily utilizes pre-trained language models to encode textual descriptions and align them with visual modalities, thereby facilitating the expression of relevant visual features. However, these approaches often struggle to establish robust alignments between fine-grained semantic concepts, leading to inconsistent representations across textual and visual information. To address these limitations, we introduce a referring remote sensing image segmentation foundational model, RSRefSeg. RSRefSeg leverages CLIP for visual and textual encoding, employing both global and local textual semantics as filters to generate referring-related visual activation features in the latent space. These activated features then serve as input prompts for SAM, which refines the segmentation masks through its robust visual generalization capabilities. Experimental results on the RRSIS-D dataset demonstrate that RSRefSeg outperforms existing methods, underscoring the effectiveness of foundational models in enhancing multimodal task comprehension. The code is available at \url{https://github.com/KyanChen/RSRefSeg}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。