首个专为遥感图像设计的开放词汇语义分割框架,解决视角和尺度变化难题。
Open-Vocabulary Remote Sensing Image Semantic Segmentation
- 基于遥感图像特性设计旋转自适应相似性计算模块
- 多尺度特征融合提升不同尺度目标的分割精度
- 首次构建遥感开放词汇分割基准,支持可复现研究
开放词汇图像语义分割(OVS)旨在对图像中任意类别进行语义区域划分。现有方法多依赖基础视觉-语言模型,通过相似性计算完成任务,但主要针对自然图像,难以应对遥感图像中快速变化的视角与显著的尺度差异。为此,本文提出首个专为遥感影像设计的OVS框架,借鉴遥感图像独特属性:为处理视角变化,引入旋转聚合相似性计算模块,生成方向自适应的初始语义图,并在空间与类别层面进行双重优化;为应对尺度剧烈变化,将多尺度图像特征融入上采样过程,生成最终的尺度感知语义掩码。为推动地球视觉领域OVS研究并促进可复现性,本文构建首个公开的遥感图像开放词汇分割基准,涵盖四个公开遥感数据集。大量实验表明,所提方法达到当前最优性能。所有代码与数据集已开源:https://github.com/caoql98/OVRS。
原文摘要 · Abstract (English)
Open-vocabulary image semantic segmentation (OVS) seeks to segment images into semantic regions across an open set of categories. Existing OVS methods commonly depend on foundational vision-language models and utilize similarity computation to tackle OVS tasks. However, these approaches are predominantly tailored to natural images and struggle with the unique characteristics of remote sensing images, such as rapidly changing orientations and significant scale variations. These challenges complicate OVS tasks in earth vision, requiring specialized approaches. To tackle this dilemma, we propose the first OVS framework specifically designed for remote sensing imagery, drawing inspiration from the distinct remote sensing traits. Particularly, to address the varying orientations, we introduce a rotation-aggregative similarity computation module that generates orientation-adaptive similarity maps as initial semantic maps. These maps are subsequently refined at both spatial and categorical levels to produce more accurate semantic maps. Additionally, to manage significant scale changes, we integrate multi-scale image features into the upsampling process, resulting in the final scale-aware semantic masks. To advance OVS in earth vision and encourage reproducible research, we establish the first open-sourced OVS benchmark for remote sensing imagery, including four public remote sensing datasets. Extensive experiments on this benchmark demonstrate our proposed method achieves state-of-the-art performance. All codes and datasets are available at https://github.com/caoql98/OVRS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。