提出双向对齐网络,提升遥感图像指代分割的精准度。
Scale-wise Bidirectional Alignment Network for Referring Remote Sensing Image Segmentation
- 设计双向对齐模块,动态融合视觉与语言特征。
- 在两个数据集上均超越现有最优方法,指标显著提升。
- 适合遥感图像理解、跨模态分割等研究者参考。
遥感图像指代分割(RRSIS)的目标是通过自然语言描述,在航拍图像中提取特定像素级区域。近年来,基于Transformer的融合设计在此领域取得显著进展。然而,现有方法主要关注在跨模态融合阶段利用语言引导优化视觉特征,忽视了视觉到语言的互补信息流,导致表征不相关或次优。此外,航拍图像中地物的空间尺度差异大,给模型在文本条件下的视觉感知带来挑战。本文提出一种新型框架——尺度双向对齐网络(SBANet),以解决上述问题。具体而言,设计带有可学习查询令牌的双向对齐模块(BAM),实现视觉与语言特征的选择性有效表示,聚焦关键令牌关联区域。BAM进一步结合动态特征选择块,同时保留宏观上下文与微观细节,促进更高效的跨模态交互。此外,SBANet引入文本条件通道与空间聚合器,弥合编码器与解码器间差距,增强复杂航拍场景中的跨尺度信息传递。大量实验表明,所提方法在RRSIS-D和RefSegRS数据集上均优于现有最先进方法,量化与定性结果均表现优异。代码将在发表后公开。
原文摘要 · Abstract (English)
The goal of referring remote sensing image segmentation (RRSIS) is to extract specific pixel-level regions within an aerial image via a natural language expression. Recent advancements, particularly Transformer-based fusion designs, have demonstrated remarkable progress in this domain. However, existing methods primarily focus on refining visual features using language-aware guidance during the cross-modal fusion stage, neglecting the complementary vision-to-language flow. This limitation often leads to irrelevant or suboptimal representations. In addition, the diverse spatial scales of ground objects in aerial images pose significant challenges to the visual perception capabilities of existing models when conditioned on textual inputs. In this paper, we propose an innovative framework called Scale-wise Bidirectional Alignment Network (SBANet) to address these challenges for RRSIS. Specifically, we design a Bidirectional Alignment Module (BAM) with learnable query tokens to selectively and effectively represent visual and linguistic features, emphasizing regions associated with key tokens. BAM is further enhanced with a dynamic feature selection block, designed to provide both macro- and micro-level visual features, preserving global context and local details to facilitate more effective cross-modal interaction. Furthermore, SBANet incorporates a text-conditioned channel and spatial aggregator to bridge the gap between the encoder and decoder, enhancing cross-scale information exchange in complex aerial scenarios. Extensive experiments demonstrate that our proposed method achieves superior performance in comparison to previous state-of-the-art methods on the RRSIS-D and RefSegRS datasets, both quantitatively and qualitatively. The code will be released after publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。