多模态融合提升恶劣天气下自动驾驶目标检测精度
SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weather
- 融合可见光、红外、雷达与激光雷达数据，动态加权适应不同天气
- 在远距离雾霾场景中，对脆弱行人检测平均精度提升17.2点
- 适用于真实复杂环境下的自动驾驶系统，尤其适合恶劣天气场景
多模态传感器融合是自主机器人在传感器失效或输入不确定时实现目标检测与决策的关键能力。尽管现有融合方法在正常环境下表现优异，但在重雾、大雪或污损导致的遮挡等恶劣天气下表现严重下降。本文提出一种面向恶劣天气的新型多传感器融合方法，除常规的RGB与LiDAR外，还引入NIR栅格相机与雷达模态，以应对低光照和恶劣天气。通过基于深度的注意力融合机制，在鸟瞰图（BEV）平面上进行特征融合与自适应优化，并采用变换器解码器根据距离与能见度动态分配模态权重。实验表明，该方法显著提升了自动驾驶车辆在极端天气下的感知可靠性，在长距离、浓雾场景下对脆弱行人的检测平均精度（AP）相比次优方法提升17.2点。
原文摘要 · Abstract (English)
Multimodal sensor fusion is an essential capability for autonomous robots, enabling object detection and decision-making in the presence of failing or uncertain inputs. While recent fusion methods excel in normal environmental conditions, these approaches fail in adverse weather, e.g., heavy fog, snow, or obstructions due to soiling. We introduce a novel multi-sensor fusion approach tailored to adverse weather conditions. In addition to fusing RGB and LiDAR sensors, which are employed in recent autonomous driving literature, our sensor fusion stack is also capable of learning from NIR gated camera and radar modalities to tackle low light and inclement weather. We fuse multimodal sensor data through attentive, depth-based blending schemes, with learned refinement on the Bird's Eye View (BEV) plane to combine image and range features effectively. Our detections are predicted by a transformer decoder that weighs modalities based on distance and visibility. We demonstrate that our method improves the reliability of multimodal sensor fusion in autonomous vehicles under challenging weather conditions, bridging the gap between ideal conditions and real-world edge cases. Our approach improves average precision by 17.2 AP compared to the next best method for vulnerable pedestrians in long distances and challenging foggy scenes. Our project page is available at https://light.princeton.edu/samfusion/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。