无需训练即可实现遥感图像开放词汇分割，提升边界精度
SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images
- 提出无训练上采样模块SimFeatUp恢复深层特征空间信息
- 通过减法操作缓解局部块令牌对全局[CLS]令牌的异常响应
- 在17个遥感数据集上平均提升4.0%~15.3%，适合低资源场景
遥感图像在农业、水资源、军事和灾后救援等领域不可或缺。像素级解析是其关键应用，但普遍受限于大量人工标注。为此，我们尝试将开放词汇语义分割（OVSS）引入遥感领域。然而，由于遥感图像对低分辨率特征敏感，预测掩码常出现目标形状扭曲、边界不匹配等问题。为此，我们提出一种简单通用的无训练上采样器SimFeatUp，以恢复深层特征中丢失的空间信息。此外，基于观察到的局部块令牌对[CLS]令牌的异常响应，我们采用直接减法操作缓解块令牌中的全局偏差。在涵盖语义分割、建筑物提取、道路检测和洪水检测任务的17个遥感数据集上进行大量实验，本方法在四项任务上分别相较现有最佳方法平均提升5.8%、8.2%、4.0%和15.3%。所有代码已开源。
原文摘要 · Abstract (English)
Remote sensing image plays an irreplaceable role in fields such as agriculture, water resources, military, and disaster relief. Pixel-level interpretation is a critical aspect of remote sensing image applications; however, a prevalent limitation remains the need for extensive manual annotation. For this, we try to introduce open-vocabulary semantic segmentation (OVSS) into the remote sensing context. However, due to the sensitivity of remote sensing images to low-resolution features, distorted target shapes and ill-fitting boundaries are exhibited in the prediction mask. To tackle this issue, we propose a simple and general upsampler, SimFeatUp, to restore lost spatial information in deep features in a training-free style. Further, based on the observation of the abnormal response of local patch tokens to [CLS] token in CLIP, we propose to execute a straightforward subtraction operation to alleviate the global bias in patch tokens. Extensive experiments are conducted on 17 remote sensing datasets spanning semantic segmentation, building extraction, road detection, and flood detection tasks. Our method achieves an average of 5.8%, 8.2%, 4.0%, and 15.3% improvement over state-of-the-art methods on 4 tasks. All codes are released. \url{https://earth-insights.github.io/SegEarth-OV}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。