提出CADFormer模型,提升遥感图像指代分割的精准度。
CADFormer: Fine-Grained Cross-modal Alignment and Decoding Transformer for Referring Remote Sensing Image Segmentation
- 设计双向细粒度对齐模块,融合视觉与语言特征
- 引入语言增强解码器,在生成掩码时利用语义上下文
- 构建高分辨率新数据集,验证复杂场景下的性能
指代遥感图像分割(RRSIS)旨在根据自然语言描述分割遥感图像中的特定目标。现有方法多采用粗粒度单向对齐,忽略语言信息在解码阶段的上下文作用,导致视觉与语言特征在目标级别对应不足,尤其在复杂表达和场景下易产生不完整或错误的掩码。为此,我们提出细粒度跨模态对齐与解码变压器CADFormer。设计语义互引导对齐模块(SMGAM),实现视觉到语言与语言到视觉的双向对齐,促进多模态特征的精细融合。进一步提出文本增强的跨模态解码器(TCMD),在解码过程中引入精细化的语言特征作为上下文,强化跨模态关系。为全面评估模型在复杂场景中对不明显目标的分割能力,我们构建了新数据集RRSIS-HR,包含更大尺寸的高分辨率遥感图像块和语义更丰富的语言描述。在RRSIS-HR与主流数据集RRSIS-D上的大量实验表明,CADFormer具有显著优势。代码与数据集将公开于https://github.com/zxk688。
原文摘要 · Abstract (English)
Referring Remote Sensing Image Segmentation (RRSIS) is a challenging task, aiming to segment specific target objects in remote sensing (RS) images based on a given language expression. Existing RRSIS methods typically employ coarse-grained unidirectional alignment approaches to obtain multimodal features, and they often overlook the critical role of language features as contextual information during the decoding process. Consequently, these methods exhibit weak object-level correspondence between visual and language features, leading to incomplete or erroneous predicted masks, especially when handling complex expressions and intricate RS image scenes. To address these challenges, we propose a fine-grained cross-modal alignment and decoding Transformer, CADFormer, for RRSIS. Specifically, we design a semantic mutual guidance alignment module (SMGAM) to achieve both vision-to-language and language-to-vision alignment, enabling comprehensive integration of visual and textual features for fine-grained cross-modal alignment. Furthermore, a textual-enhanced cross-modal decoder (TCMD) is introduced to incorporate language features during decoding, using refined textual information as context to enhance the relationship between cross-modal features. To thoroughly evaluate the performance of CADFormer, especially for inconspicuous targets in complex scenes, we constructed a new RRSIS dataset, called RRSIS-HR, which includes larger high-resolution RS image patches and semantically richer language expressions. Extensive experiments on the RRSIS-HR dataset and the popular RRSIS-D dataset demonstrate the effectiveness and superiority of CADFormer. Datasets and source codes will be available at https://github.com/zxk688.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。