将遥感图像指代分割任务拆解为粗定位与精分割两阶段,提升精度与可解释性。
RSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Models
- 采用双阶段解耦架构,先用CLIP粗定位,再用SAM精分割。
- 在多个遥感数据集上实现约3%的gIoU提升,复杂语义理解更准确。
- 适合需要高精度遥感分析的科研与应用人员使用。
指代遥感图像分割提供了一种通过视觉-语言协同解析实现灵活且细粒度的遥感场景分析框架。现有方法多采用三阶段流水线:双模态编码、跨模态交互和像素解码。这些方法在处理复杂语义关系和实现精确跨模态对齐方面存在显著局限,主要源于目标定位与边界划分耦合的处理机制,导致语义模糊时误差传播加剧,模型泛化性和可解释性受限。为此,本文提出RSRefSeg 2,一种解耦范式,将传统流程重构为协作式双阶段框架:粗定位后接精细分割。该方法通过战略性地融合CLIP的跨模态对齐能力与SAM的分割泛化能力。具体而言,使用CLIP作为双模态编码器,在其预对齐的语义空间中激活目标特征并生成定位提示。为缓解在多实体场景下引用文本引起的CLIP误激活问题,设计了级联二阶提示器,通过将文本嵌入分解至互补语义子空间,实现隐式推理以提升精度。优化后的语义提示随后引导SAM生成像素级细化掩码,完成语义传递流程。大量实验(RefSegRS、RRSIS-D 和 RISBench)表明,RSRefSeg 2 在分割精度(+~3% gIoU)和复杂语义解析能力上超越当前主流方法。代码已开源:https://github.com/KyanChen/RSRefSeg2。
原文摘要 · Abstract (English)
Referring Remote Sensing Image Segmentation provides a flexible and fine-grained framework for remote sensing scene analysis via vision-language collaborative interpretation. Current approaches predominantly utilize a three-stage pipeline encompassing dual-modal encoding, cross-modal interaction, and pixel decoding. These methods demonstrate significant limitations in managing complex semantic relationships and achieving precise cross-modal alignment, largely due to their coupled processing mechanism that conflates target localization with boundary delineation. This architectural coupling amplifies error propagation under semantic ambiguity while restricting model generalizability and interpretability. To address these issues, we propose RSRefSeg 2, a decoupling paradigm that reformulates the conventional workflow into a collaborative dual-stage framework: coarse localization followed by fine segmentation. RSRefSeg 2 integrates CLIP's cross-modal alignment strength with SAM's segmentation generalizability through strategic foundation model collaboration. Specifically, CLIP is employed as the dual-modal encoder to activate target features within its pre-aligned semantic space and generate localization prompts. To mitigate CLIP's misactivation challenges in multi-entity scenarios described by referring texts, a cascaded second-order prompter is devised, which enhances precision through implicit reasoning via decomposition of text embeddings into complementary semantic subspaces. These optimized semantic prompts subsequently direct the SAM to generate pixel-level refined masks, thereby completing the semantic transmission pipeline. Extensive experiments (RefSegRS, RRSIS-D, and RISBench) demonstrate that RSRefSeg 2 surpasses contemporary methods in segmentation accuracy (+~3% gIoU) and complex semantic interpretation. Code is available at: https://github.com/KyanChen/RSRefSeg2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。