首个针对多模态分割鲁棒性的评测基准,评估传感器失效下的性能表现。
Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness
- 构建三种故障场景:模态全缺、随机缺失与噪声输入,模拟真实环境不确定性。
- 提出四类指标,量化模型在模态缺失或噪声下的平均与极端性能表现。
- 开源代码并系统梳理现有方法,为实际部署提供可靠评估工具。
多模态语义分割(MMSS)通过融合不同模态的互补信息,克服单模态数据的局限性。尽管进展显著,研究与实际应用之间仍存在差距,主要源于多模态数据质量的可变性与不确定性。鲁棒性已成为实用化MMSS的关键。然而,缺乏标准化的鲁棒性评测基准制约了进一步发展。为此,本文首先综述现有MMSS文献,对代表性方法进行分类归纳。随后,构建一个鲁棒性评测基准,评估模型在三种场景下的表现:模态全缺(EMM)、随机缺失(RMM)和噪声模态(NM)。从概率角度建模模态故障,考虑两种情形:(1)所有损坏组合等概率;(2)各模态独立按伯努利分布失效。基于此,提出四个指标——$mIoU^{Avg}_{EMM}$、$mIoU^{E}_{EMM}$、$mIoU^{Avg}_{RMM}$、$mIoU^{E}_{RMM}$,用于评估模型在EMM和RMM下的鲁棒性。本工作首次提供专门针对MMSS鲁棒性的评测基准,为领域发展提供新洞见与工具。源码已公开于 https://github.com/Chenfei-Liao/Multi-Modal-Semantic-Segmentation-Robustness-Benchmark。
原文摘要 · Abstract (English)
Multi-modal semantic segmentation (MMSS) addresses the limitations of single-modality data by integrating complementary information across modalities. Despite notable progress, a significant gap persists between research and real-world deployment due to variability and uncertainty in multi-modal data quality. Robustness has thus become essential for practical MMSS applications. However, the absence of standardized benchmarks for evaluating robustness hinders further advancement. To address this, we first survey existing MMSS literature and categorize representative methods to provide a structured overview. We then introduce a robustness benchmark that evaluates MMSS models under three scenarios: Entire-Missing Modality (EMM), Random-Missing Modality (RMM), and Noisy Modality (NM). From a probabilistic standpoint, we model modality failure under two conditions: (1) all damaged combinations are equally probable; (2) each modality fails independently following a Bernoulli distribution. Based on these, we propose four metrics-$mIoU^{Avg}_{EMM}$, $mIoU^{E}_{EMM}$, $mIoU^{Avg}_{RMM}$, and $mIoU^{E}_{RMM}$-to assess model robustness under EMM and RMM. This work provides the first dedicated benchmark for MMSS robustness, offering new insights and tools to advance the field. Source code is available at https://github.com/Chenfei-Liao/Multi-Modal-Semantic-Segmentation-Robustness-Benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。