统一处理雷达转光学图像与语义分割,提升遥感图像可解释性。
Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation

- 用共享的分层ViT模型协同优化图像转换与语义分割。
- 在有/无配对数据下均实现高质量图像生成与精准分割。
- 适合遥感图像分析、多模态融合等应用研究者使用。
合成孔径雷达(SAR)具备全天候、全天时观测能力,但其斑点噪声和非直观的散射机制限制了图像可读性。现有的SAR到光学图像(S2O)转换生成模型常为追求视觉效果忽略语义结构约束,影响下游任务。本文提出统一协同双任务学习框架BMT(Bridging Modalities and Tasks),通过共享分层视觉变换器联合优化S2O图像转换与语义分割。该框架包含:(1) 结合全局自注意力与空间深度卷积的LocalViTBlock,通过可学习门控机制融合;(2) 多尺度细化、色彩校正与抗混叠的增强输出模块,通过特征融合校准通道级色彩统计;(3) 类ControlNet的条件注入机制,将SAR小波特征与分割标签编码为多尺度特征金字塔,并在每层编码器中通过零初始化卷积注入;(4) 有界Kendall不确定性加权策略,防止任一任务主导共享表征。在公开的WHU-OPT-SAR配对数据集及基于HRSID和DIOR构建的自建非配对船舶数据集上进行评估。实验表明,所提方法在配对与非配对设置下均取得具有竞争力的S2O图像生成质量与语义分割性能。代码与数据集已开源:https://github.com/Lewisyuaner/BMT-S2O-main。
原文摘要 · Abstract (English)
Synthetic Aperture Radar (SAR) images have all-weather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images. Generative models for SAR-to-optical (S2O) conversion can improve visual interpretability, but existing methods often ignore the constraints on semantic structure, which are necessary for downstream tasks, for the sake of visual effects. We propose a unified collaborative dual-task learning framework, termed BMT (Bridging Modalities and Tasks), that jointly optimizes S2O image translation and semantic segmentation through a shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale refinement processing, color correction and anti-aliasing, which calibrates channel-level color statistics through feature fusion; (3) a ControlNet-style conditional injection mechanism that encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolution; (4) a bounded Kendall uncertainty weighting scheme that prevents either task from dominating the shared representation. We evaluate the framework under both paired and unpaired translation settings, on the public WHU-OPT-SAR paired dataset and a self-constructed unpaired ship dataset built from HRSID and DIOR, respectively. The experimental results show that the proposed method achieves competitive S2O translation quality and semantic segmentation performance. The dataset and source code have been publicly released at https://github.com/Lewisyuaner/BMT-S2O-main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。