提出光照感知的多模态融合网络，提升自动驾驶路况感知鲁棒性
A Multi-modal Fusion Network for Terrain Perception Based on Illumination Aware
- 基于光照特征动态调整相机与雷达数据融合权重
- 在复杂光照下路面识别准确率显著优于单模态方法
- 适合自动驾驶系统在夜间或雨天等极端条件下的路况感知
道路地形对自动驾驶车辆的行驶安全至关重要。然而，现有传感器（如摄像头和激光雷达）易受光照和天气变化影响，难以实现实时路况感知。本文提出一种光照感知的多模态融合网络（IMF），结合外感受与本体感知信息，并根据光照特征优化融合过程。设计光照感知子网络以精确估计光照特征，构建可动态调整各模态权重的多模态融合网络。通过预训练光照感知子网络并引入光照损失作为训练约束，进一步优化性能。大量实验表明，IMF在多种光照条件下均优于当前最优方法。与单模态方法的对比凸显了多模态融合在复杂光照下精准感知路面的优势。数据集已公开：https://github.com/lindawang2016/IMF。
原文摘要 · Abstract (English)
Road terrains play a crucial role in ensuring the driving safety of autonomous vehicles (AVs). However, existing sensors of AVs, including cameras and Lidars, are susceptible to variations in lighting and weather conditions, making it challenging to achieve real-time perception of road conditions. In this paper, we propose an illumination-aware multi-modal fusion network (IMF), which leverages both exteroceptive and proprioceptive perception and optimizes the fusion process based on illumination features. We introduce an illumination-perception sub-network to accurately estimate illumination features. Moreover, we design a multi-modal fusion network which is able to dynamically adjust weights of different modalities according to illumination features. We enhance the optimization process by pre-training of the illumination-perception sub-network and incorporating illumination loss as one of the training constraints. Extensive experiments demonstrate that the IMF shows a superior performance compared to state-of-the-art methods. The comparison results with single modality perception methods highlight the comprehensive advantages of multi-modal fusion in accurately perceiving road terrains under varying lighting conditions. Our dataset is available at: https://github.com/lindawang2016/IMF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。