提出新框架提升多模态分割在传感器缺失时的鲁棒性,同时保持全模态下的高性能。
RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selection
- 分两阶段训练:先跨模态知识蒸馏增强鲁棒性,再通过特征选择模块优化表示
- 在三个数据集上缺失模态性能提升0.89%~3.89%,全模态下仅损失0.1%精度
- 适用于实际部署中传感器不稳定场景,适配多种骨干网络
多模态语义分割在真实应用中常因传感器数据不完整、退化或缺失而面临挑战。现有方法多采用模态丢弃自蒸馏提升鲁棒性,但忽视模态间相关性,导致无模态缺失时性能显著下降。为此,本文提出RMMSS,一种两阶段框架,在缺失模态条件下逐步提升模型鲁棒性,同时维持全模态下的强性能。其包含两个核心组件:混合原型蒸馏模块(HPDM)与特征选择模块(FSM)。第一阶段使用全模态数据预训练教师模型,并引入HPDM实现跨模态知识蒸馏,获得高鲁棒性模型;第二阶段冻结预训练教师模型与鲁棒模型,提出可训练的FSM,通过特征评分从两模型的特征与logits层提取最优表示,学习最终学生模型。该模型在保持强鲁棒性的同时,实现全模态下的高精度表现。在三个数据集上的实验表明,相比当前最优方法,本方法在缺失模态下性能提升2.80%、3.89%和0.89%,而全模态下仅降低0.1% mIoU。不同骨干网络(AnySeg与CMNeXt)验证了框架的泛化能力。
原文摘要 · Abstract (English)
Multi-modal semantic segmentation (MMSS) faces significant challenges in real-world applications due to incomplete, degraded, or missing sensor data. While current MMSS methods typically use self-distillation with modality dropout to improve robustness, they largely overlook inter-modal correlations and thus suffer significant performance degradation when no modalities are missing. To this end, we present RMMSS, a two-stage framework designed to progressively enhance model robustness under missing-modality conditions, while maintaining strong performance in full-modality scenarios. It comprises two key components: the Hybrid Prototype Distillation Module (HPDM) and the Feature Selection Module (FSM). In the first stage, we pre-train the teacher model with full-modality data and then introduce HPDM to do cross-modal knowledge distillation for obtaining a highly robust model. In the second stage, we freeze both the pre-trained full-modality teacher model and the robust model and propose a trainable FSM that extracts optimal representations from both the feature and logits layers of the models via feature score calculation. This process learns a final student model that maintains strong robustness while achieving high performance under full-modality conditions. Our experiments on three datasets demonstrate that our method improves missing-modality performance by 2.80%, 3.89%, and 0.89%, respectively, compared to the state-of-the-art, while causing almost no drop in full-modality performance (only -0.1% mIoU). Meanwhile, different backbones (AnySeg and CMNeXt) are utilized to validate the generalizability of our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。