通过边缘线索统一事件与图像特征,提升极端条件下的分割鲁棒性。
Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation
- 利用边缘信息重编码融合事件与图像特征,解决模态异构问题。
- 在合成和真实数据集上实现2.55% mIoU提升,抗遮挡能力更强。
- 适合做多模态感知、极端环境视觉任务的研究者参考。
语义分割在理想条件下表现优异,但在极端场景(如弱光、剧烈相机运动)下,RGB信息严重丢失,导致分割性能下降。现有方法尝试引入高速高动态的事件模态作为补充,但事件与RGB天然异构,造成特征不匹配,现有融合方法优化效果不佳。不同于以往工作,本文深入挖掘双模态的边缘特性,提出一种新的边缘感知语义一致性框架(Edge-awareness Semantic Concordance, ESC),通过潜在边缘线索统一异构特征。首先提出边缘感知潜在重编码机制,在重构事件-图像特征至统一语义空间时生成不确定性指标,并借助预设边缘词典将分布转化为重编码特征。随后设计重编码融合与不确定性优化模块,有效解决极端条件下的异构融合问题。构建了两个合成数据集和一个真实世界事件-图像语义分割数据集用于极端场景对比。实验表明,本方法在自建数据集DERS-XS上相比当前最优方法提升2.55% mIoU,且在空间遮挡下表现出更强鲁棒性。代码与数据集已开源。
原文摘要 · Abstract (English)
Semantic segmentation has achieved great success in ideal conditions. However, when facing extreme conditions (e.g., insufficient light, fierce camera motion), most existing methods suffer from significant information loss of RGB, severely damaging segmentation results. Several researches exploit the high-speed and high-dynamic event modality as a complement, but event and RGB are naturally heterogeneous, which leads to feature-level mismatch and inferior optimization of existing multi-modality methods. Different from these researches, we delve into the edge secret of both modalities for resilient fusion and propose a novel Edge-awareness Semantic Concordance framework to unify the multi-modality heterogeneous features with latent edge cues. In this framework, we first propose Edge-awareness Latent Re-coding, which obtains uncertainty indicators while realigning event-RGB features into unified semantic space guided by re-coded distribution, and transfers event-RGB distributions into re-coded features by utilizing a pre-established edge dictionary as clues. We then propose Re-coded Consolidation and Uncertainty Optimization, which utilize re-coded edge features and uncertainty indicators to solve the heterogeneous event-RGB fusion issues under extreme conditions. We establish two synthetic and one real-world event-RGB semantic segmentation datasets for extreme scenario comparisons. Experimental results show that our method outperforms the state-of-the-art by a 2.55% mIoU on our proposed DERS-XS, and possesses superior resilience under spatial occlusion. Our code and datasets are publicly available at https://github.com/iCVTEAM/ESC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。