让深度相机失效时，模型仍能靠颜色图准确分割场景
Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropout
- 训练时随机屏蔽颜色或深度数据，模拟传感器故障
- 在NYU-Depth V2和SUN RGB-D上缺失模态下准确率提升12%以上
- 适合部署在摄像头或深度传感器易损坏的实时场景
RGB-D语义分割虽有显著进展，但多数模型假设彩色与深度数据始终可用。实际中监控传感器故障或遮挡常导致一模态缺失。尽管单模态仍含足够信息，但仅用完整数据训练的模型在缺模态时性能严重下降。为此提出简单持续训练方法「条件丢弃（ConD）」：从预训练的RGB-D模型出发，增加第二阶段训练，随机生成全模态、无颜色、无深度三种输入，冻结原编码器，对复制编码器采用零初始化特征注入进行训练。在NYU-Depth V2和SUN RGB-D数据集上的实验表明，ConD显著提升缺失模态下的鲁棒性，甚至在模态完整时也带来轻微性能提升。代码将在论文录用后公开。
原文摘要 · Abstract (English)
RGB-D semantic segmentation has achieved remarkable progress, yet most models assume that RGB and depth are always available. In practice, failures or occlusions of surveillance sensors often remove one modality. Although RGB or depth alone can contain sufficient cues, models trained only on full-modality inputs fail to exploit the remaining modality once one is missing, causing severe degradation. We tackle this issue with a simple continued-training paradigm, \emph{Condition Dropout (ConD)}, which mitigates degradation while preserving full-modality accuracy. Starting from a pretrained RGB-D model, ConD adds a second stage that randomly simulates complete, RGB-missing, and depth-missing inputs, freezes the original encoders, and trains copied encoders with zero-initialized feature injection. Experiments on NYU-Depth V2 and SUN RGB-D show that ConD improves robustness under missing modalities and even yields slight gains when modalities are complete. Our code will be made publicly available upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。