根据环境条件动态融合多传感器数据，提升自动驾驶场景感知鲁棒性。
CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes
- 用条件令牌引导多模态融合，按实际环境调整策略
- 在MUSES上达59.7 PQ（全景分割）和78.2 mIoU（语义分割）
- 适合需要高鲁棒性的自动驾驶感知系统
多传感器融合对自动驾驶中稳健的语义感知至关重要，因各类传感器具有互补优劣。然而现有方法通常对所有条件一视同仁，导致性能不佳。本文提出一种新型条件感知多模态融合方法CAFuser，利用RGB相机分类环境条件并生成条件令牌，指导多模态融合。我们引入模态特定特征适配器，将不同传感器输入对齐至共享潜在空间，实现与单一预训练主干网络的高效集成。通过根据实际条件动态调整融合策略，模型显著提升在恶劣条件下的鲁棒性与精度。CAFuser在公开基准MUSES上排名第一，实现59.7 PQ（多模态全景分割）和78.2 mIoU（语义分割），并在DeLiVER上达到新SOTA。代码已开源：https://github.com/timbroed/CAFuser。
原文摘要 · Abstract (English)
Leveraging multiple sensors is crucial for robust semantic perception in autonomous driving, as each sensor type has complementary strengths and weaknesses. However, existing sensor fusion methods often treat sensors uniformly across all conditions, leading to suboptimal performance. By contrast, we propose a novel, condition-aware multimodal fusion approach for robust semantic perception of driving scenes. Our method, CAFuser, uses an RGB camera input to classify environmental conditions and generate a Condition Token that guides the fusion of multiple sensor modalities. We further newly introduce modality-specific feature adapters to align diverse sensor inputs into a shared latent space, enabling efficient integration with a single and shared pre-trained backbone. By dynamically adapting sensor fusion based on the actual condition, our model significantly improves robustness and accuracy, especially in adverse-condition scenarios. CAFuser ranks first on the public MUSES benchmarks, achieving 59.7 PQ for multimodal panoptic and 78.2 mIoU for semantic segmentation, and also sets the new state of the art on DeLiVER. The source code is publicly available at: https://github.com/timbroed/CAFuser.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。