arXiv:2412.19492cs.CVcs.MM2024-12AAAI被引 48

让遥感图像分割能识别任意新类别,突破传统模型的封闭词表限制。

Towards Open-Vocabulary Remote Sensing Image Semantic Segmentation

  • 融合遥感专用与通用视觉语言模型,构建双流特征提取框架。
  • 在51,846张图像上实现40类开放词汇分割,性能显著超越现有方法。
  • 适合需要灵活适应新地物类别的遥感分析、城市规划等应用。

近年来,基于深度学习的遥感图像分割方法取得了显著进展,但通常依赖预定义的语义类别,需额外标注和训练才能新增类别,且无法分割任意语义类。为此,本文提出开放词汇遥感图像语义分割(OVRSISS),旨在对任意语义类别进行分割。为解决数据缺失问题,我们构建了包含51,846张图像、覆盖40种多样语义类别的LandDiscover50K数据集。同时提出新型框架GSNet,融合遥感专用模型领域先验与通用视觉-语言模型的泛化能力。GSNet包含双流图像编码器(DSIE)、查询引导特征融合(QGFF)和残差信息保留解码器(RIPD)。DSIE在双通道中分别捕获专用与通用模型的全面特征;QGFF在可变词汇引导下融合专精与通用特征,实现互补;RIPD聚合多源特征以提升掩码预测精度。实验表明,该方法大幅优于现有方法,所提LandDiscover50K也有效提升OVRSISS性能。相关代码与数据将公开于https://github.com/yecy749/GSNet。

原文摘要 · Abstract (English)

Recently, deep learning based methods have revolutionized remote sensing image segmentation. However, these methods usually rely on a pre-defined semantic class set, thus needing additional image annotation and model training when adapting to new classes. More importantly, they are unable to segment arbitrary semantic classes. In this work, we introduce Open-Vocabulary Remote Sensing Image Semantic Segmentation (OVRSISS), which aims to segment arbitrary semantic classes in remote sensing images. To address the lack of OVRSISS datasets, we develop LandDiscover50K, a comprehensive dataset of 51,846 images covering 40 diverse semantic classes. In addition, we propose a novel framework named GSNet that integrates domain priors from special remote sensing models and versatile capabilities of general vision-language models. Technically, GSNet consists of a Dual-Stream Image Encoder (DSIE), a Query-Guided Feature Fusion (QGFF), and a Residual Information Preservation Decoder (RIPD). DSIE first captures comprehensive features from both special models and general models in dual streams. Then, with the guidance of variable vocabularies, QGFF integrates specialist and generalist features, enabling them to complement each other. Finally, RIPD is proposed to aggregate multi-source features for more accurate mask predictions. Experiments show that our method outperforms other methods by a large margin, and our proposed LandDiscover50K improves the performance of OVRSISS methods. The proposed dataset and method will be made publicly available at https://github.com/yecy749/GSNet.

遥感分割开放词汇视觉语言模型多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。