arXiv:2603.02505cs.CV2026-03被引 1

解决遥感多模态数据缺失下的分割难题,提升弱模态表现

SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data

  • 引入语义引导的融合与模态感知采样,动态平衡各模态贡献
  • 在多个数据集上显著提升弱模态分割精度,最高增益达4.2% mIoU
  • 适合处理传感器故障或覆盖不全的遥感影像分析任务

多模态语义分割通过融合不同传感器的互补信息用于遥感地球观测。然而实际系统常因传感器故障或覆盖不全导致模态缺失,即不完整多模态语义分割(IMSS)。IMSS面临三大挑战:(1) 多模态不平衡,主导模态压制脆弱模态;(2) 模态间类内差异大,尺度、形状、朝向变化显著;(3) 跨模态异质性,冲突线索引发不一致语义响应。现有方法依赖对比学习或联合优化,易过度对齐,丢失模态特异性信息或训练失衡,偏向强模态,且忽略类内差异与跨模态异质性。为此,提出语义引导的模态感知(SGMA)框架,通过语义引导实现均衡多模态学习,减少类内差异并调和跨模态不一致性。SGMA包含两个即插即用模块:(1) 语义引导融合(SGF)模块提取多尺度、类别级语义原型,捕捉跨模态一致类别表征,基于原型-特征对齐估计各模态鲁棒性,并按鲁棒性加权自适应融合,缓解类内差异与跨模态异质性;(2) 模态感知采样(MAS)模块利用SGF的鲁棒性估计动态重加权训练样本,优先选择脆弱模态中的难样本,解决模态不平衡问题。在多个数据集和骨干网络上的大量实验表明,SGMA持续优于当前最优方法,尤其在脆弱模态上提升显著。

原文摘要 · Abstract (English)

Multimodal semantic segmentation integrates complementary information from diverse sensors for remote sensing Earth observation. However, practical systems often encounter missing modalities due to sensor failures or incomplete coverage, termed Incomplete Multimodal Semantic Segmentation (IMSS). IMSS faces three key challenges: (1) multimodal imbalance, where dominant modalities suppress fragile ones; (2) intra-class variation in scale, shape, and orientation across modalities; and (3) cross-modal heterogeneity with conflicting cues producing inconsistent semantic responses. Existing methods rely on contrastive learning or joint optimization, which risk over-alignment, discarding modality-specific cues or imbalanced training, favoring robust modalities, while largely overlooking intra-class variation and cross-modal heterogeneity. To address these limitations, we propose the Semantic-Guided Modality-Aware (SGMA) framework, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance. SGMA introduces two complementary plug-and-play modules: (1) Semantic-Guided Fusion (SGF) module extracts multi-scale, class-wise semantic prototypes that capture consistent categorical representations across modalities, estimates per-modality robustness based on prototype-feature alignment, and performs adaptive fusion weighted by robustness scores to mitigate intra-class variation and cross-modal heterogeneity; (2) Modality-Aware Sampling (MAS) module leverages robustness estimations from SGF to dynamically reweight training samples, prioritizing challenging samples from fragile modalities to address modality imbalance. Extensive experiments across multiple datasets and backbones demonstrate that SGMA consistently outperforms state-of-the-art methods, with particularly significant improvements in fragile modalities.

遥感分割多模态学习语义引导数据缺失

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。