arXiv:2604.15856cs.CVcs.AI2026-04被引 1

提出结构化隐空间投影,让多光谱分割在缺模态时仍保持精准。

Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection

论文配图:Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
图 1 · 摘自论文原文
  • 将隐空间分为共享与特有成分,动态适配缺失模态。
  • 在三组遥感数据上,全模态与缺模态下均超越现有模型。
  • 适合需要应对传感器故障的遥感图像分割场景。

多模态遥感数据为语义分割提供互补信息,但实际部署中因传感器故障、采集问题或恶劣大气条件可能导致部分模态缺失。现有方法通常通过学习共享表示来处理缺失模态,但这会牺牲模态特异性信息,导致全模态时性能下降。本文提出CBC-SLP模型,旨在同时保留模态不变与模态特异性信息。受模态对齐理论启发——完全对齐的多模态表示可能损害下游任务表现——我们设计了一种结构化隐空间投影架构,作为模型的归纳偏置,不依赖损失函数强制。具体地,将隐空间划分为共享与模态特异成分,并根据随机模态可用掩码自适应传递至解码器。在三个多光谱遥感图像数据集上的实验表明,CBC-SLP在全模态与缺失模态场景下均持续优于当前最优模型。此外,实证显示该策略能恢复共享表示中未保留的互补信息。代码已公开于https://github.com/iremulku/Multispectral-Semantic-Segmentation-via-Structured-Latent-Projection-CBC-SLP-。

原文摘要 · Abstract (English)

Multimodal remote sensing data provide complementary information for semantic segmentation, but in real-world deployments, some modalities may be unavailable due to sensor failures, acquisition issues, or challenging atmospheric conditions. Existing multimodal segmentation models typically address missing modalities by learning a shared representation across inputs. However, this approach can introduce a trade-off by compromising modality-specific complementary information and reducing performance when all modalities are available. In this paper, we tackle this limitation with CBC-SLP, a multimodal semantic segmentation model designed to preserve both modality-invariant and modality-specific information. Inspired by the theoretical results on modality alignment, which state that perfectly aligned multimodal representations can lead to sub-optimal performance in downstream prediction tasks, we propose a novel structured latent projection approach as an architectural inductive bias. Rather than enforcing this strategy through a loss term, we incorporate it directly into the architecture. In particular, to use the complementary information effectively while maintaining robustness under random modality dropout, we structure the latent representations into shared and modality-specific components and adaptively transfer them to the decoder according to the random modality availability mask. Extensive experiments on three multimodal remote sensing image sets demonstrate that CBC-SLP consistently outperforms state-of-the-art multimodal models across full and missing modality scenarios. Besides, we empirically demonstrate that the proposed strategy can recover the complementary information that may not be preserved in a shared representation. The code is available at https://github.com/iremulku/Multispectral-Semantic-Segmentation-via-Structured-Latent-Projection-CBC-SLP-.

遥感分割多模态鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。