用提示词增强的SAM模型,让遥感图像分割跨域更准更快。
Joint-Optimized Unsupervised Adversarial Domain Adaptation in Remote Sensing Segmentation with Prompted Foundation Model
- 用SAM提取通用特征,再通过提示词生成无类别图引导优化
- 在Potsdam/Vaihingen等数据集上达新高,最高提升4.2% mIoU
- 适合做遥感图像分割但标注少、场景变化大的研究者
无监督域适应遥感语义分割(UDA-RSSeg)旨在将源域训练的模型适配到目标域,减少对标注数据的依赖。该任务面临两大挑战:不同遥感域间特征表示差异大,以及源域模式偏差导致特征转为预测概率时产生域差距。为此,我们提出结合“通用图像分割模型(SAM)”的联合优化对抗网络(SAM-JOANet)。该方法利用SAM的强泛化能力缓解特征不一致问题,设计微调解码器将SAM编码器特征转换为预测结果,并引入基于特征层级对抗的提示分割器生成类无关地图,指导解码器优化。整个网络端到端联合训练。在ISPRS(Potsdam/Vaihingen)和CITY-OSM(Paris/Chicago)等基准数据集上的大量实验表明,该方法有效且鲁棒,可视化分析证实其可解释性。代码已开源。
原文摘要 · Abstract (English)
Unsupervised Domain Adaptation for Remote Sensing Semantic Segmentation (UDA-RSSeg) addresses the challenge of adapting a model trained on source domain data to target domain samples, thereby minimizing the need for annotated data across diverse remote sensing scenes. This task presents two principal challenges: (1) severe inconsistencies in feature representation across different remote sensing domains, and (2) a domain gap that emerges due to the representation bias of source domain patterns when translating features to predictive logits. To tackle these issues, we propose a joint-optimized adversarial network incorporating the "Segment Anything Model (SAM) (SAM-JOANet)" for UDA-RSSeg. Our approach integrates SAM to leverage its robust generalized representation capabilities, thereby alleviating feature inconsistencies. We introduce a finetuning decoder designed to convert SAM-Encoder features into predictive logits. Additionally, a feature-level adversarial-based prompted segmentor is employed to generate class-agnostic maps, which guide the finetuning decoder's feature representations. The network is optimized end-to-end, combining the prompted segmentor and the finetuning decoder. Extensive evaluations on benchmark datasets, including ISPRS (Potsdam/Vaihingen) and CITY-OSM (Paris/Chicago), demonstrate the effectiveness of our method. The results, supported by visualization and analysis, confirm the method's interpretability and robustness. The code of this paper is available at https://github.com/CV-ShuchangLyu/SAM-JOANet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。