提出GEMMNet模型,用生成式方法补全遥感图像缺失模态,提升分割精度。
Filling the Gaps: A Multitask Hybrid Multiscale Generative Framework for Missing Modality in Remote Sensing Semantic Segmentation
- 设计混合特征提取与多尺度融合机制,更好捕捉遥感数据语义信息。
- 在Vaihingen和Potsdam数据集上优于AE、cGAN等生成模型及mmformer等非生成模型。
- 适合处理传感器故障或恶劣天气导致的遥感多模态数据缺失问题。
多模态学习在多个领域显著优于单模态模型,但在真实场景中,由于传感器故障或恶劣天气,多模态信号常出现缺失,严重降低模型性能。生成模型如自编码器(AE)和生成对抗网络(GAN)可尝试从已有模态重建缺失模态,但其在遥感语义分割中的效果尚未充分探索。本文分析现有生成方法在处理遥感多模态数据异质性时的局限:难以捕捉复杂场景中类内差异大、类间差异小的语义上下文;且传统生成模型过度依赖主导模态,引入偏差,影响缺失模态下的鲁棒性。为此,我们提出生成增强型多模态网络GEMMNet,包含三个核心组件:(1) 混合特征提取器(HyFEx),有效学习模态特异性表示;(2) 多尺度感知的混合融合(HyFMA),跨尺度捕获模态协同语义上下文;(3) 补偿损失(CoLoss)机制,通过促进模态与任务间一致性缓解固有偏差。GEMMNet在两个具有挑战性的遥感语义分割数据集(Vaihingen 和 Potsdam)上,超越了AE、cGAN等生成基线以及mmformer、shaspec等先进非生成方法。代码已开源。
原文摘要 · Abstract (English)
Multimodal learning has shown significant performance boost compared to ordinary unimodal models across various domains. However, in real-world scenarios, multimodal signals are susceptible to missing because of sensor failures and adverse weather conditions, which drastically deteriorates models' operation and performance. Generative models such as AutoEncoder (AE) and Generative Adversarial Network (GAN) are intuitive solutions aiming to reconstruct missing modality from available ones. Yet, their efficacy in remote sensing semantic segmentation remains underexplored. In this paper, we first examine the limitations of existing generative approaches in handling the heterogeneity of multimodal remote sensing data. They inadequately capture semantic context in complex scenes with large intra-class and small inter-class variation. In addition, traditional generative models are susceptible to heavy dependence on the dominant modality, introducing bias that affects model robustness under missing modality conditions. To tackle these limitations, we propose a novel Generative-Enhanced MultiModal learning Network (GEMMNet) with three key components: (1) Hybrid Feature Extractor (HyFEx) to effectively learn modality-specific representations, (2) Hybrid Fusion with Multiscale Awareness (HyFMA) to capture modality-synergistic semantic context across scales and (3) Complementary Loss (CoLoss) scheme to alleviate the inherent bias by encouraging consistency across modalities and tasks. Our method, GEMMNet, outperforms both generative baselines AE, cGAN (conditional GAN), and state-of-the-art non-generative approaches - mmformer and shaspec - on two challenging semantic segmentation remote sensing datasets (Vaihingen and Potsdam). Source code is made available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。