arXiv:2603.07486cs.CV2026-03被引 2

通过解耦与重构多模态特征,提升传感器损坏下的3D目标检测鲁棒性。

Multi-Modal Decouple and Recouple Network for Robust 3D Object Detection

  • 将相机与激光雷达的BEV特征解耦为共性与特有部分,实现跨模态补偿
  • 设计三路专家网络分别应对激光雷达、相机或双模态损坏,性能更稳
  • 在nuScenes数据集上验证,对真实世界数据污染场景效果最佳

基于鸟瞰图(BEV)的多模态3D目标检测在基准测试中已取得显著进展。然而,在真实场景中,由于传感器配置问题(如激光雷达)或环境条件变化(如相机)导致的数据污染,检测精度可能大幅下降。此前模型存在多模态BEV特征融合过于紧密的问题,一旦某一模态或双模态受损,整体性能易受拖累。为此,本文提出一种多模态解耦与重构网络,以增强在数据污染下的鲁棒性。我们观察到不同模态共享一些高层不变特征,且这些特征不会同时失效,因不同类型污染对各模态影响方式不同。因此,可通过跨模态恢复这些不变特征实现稳健融合。具体地,我们将相机和激光雷达的BEV特征显式解耦为模态不变和模态特异部分,使不变特征可相互补偿,减轻单模态损坏的负面影响。随后,将特征重构成三个专家网络,分别处理激光雷达损坏、相机损坏及双模态损坏的情况:每个专家利用不变特征作为鲁棒信息,特异特征作为补充。最后,自适应融合三路专家输出,生成用于3D目标检测的鲁棒特征。为验证,我们在nuScenes基础上构建了一个包含大量激光雷达、相机及双模态污染的基准数据集。模型在干净nuScenes上训练,于各类污染条件下测试,结果表明,无论在污染还是无污染数据上,均优于近期主流模型。

原文摘要 · Abstract (English)

Multi-modal 3D object detection with bird's eye view (BEV) has achieved desired advances on benchmarks. Nonetheless, the accuracy may drop significantly in the real world due to data corruption such as sensor configurations for LiDAR and scene conditions for camera. One design bottleneck of previous models resides in the tightly coupling of multi-modal BEV features during fusion, which may degrade the overall system performance if one modality or both is corrupted. To mitigate, we propose a Multi-Modal Decouple and Recouple Network for robust 3D object detection under data corruption. Different modalities commonly share some high-level invariant features. We observe that these invariant features across modalities do not always fail simultaneously, because different types of data corruption affect each modality in distinct ways.These invariant features can be recovered across modalities for robust fusion under data corruption.To this end, we explicitly decouple Camera/LiDAR BEV features into modality-invariant and modality-specific parts. It allows invariant features to compensate each other while mitigates the negative impact of a corrupted modality on the other.We then recouple these features into three experts to handle different types of data corruption, respectively, i.e., LiDAR, camera, and both.For each expert, we use modality-invariant features as robust information, while modality-specific features serve as a complement.Finally, we adaptively fuse the three experts to exact robust features for 3D object detection. For validation, we collect a benchmark with a large quantity of data corruption for LiDAR, camera, and both based on nuScenes. Our model is trained on clean nuScenes and tested on all types of data corruption. Our model consistently achieves the best accuracy on both corrupted and clean data compared to recent models.

3D检测多模态鲁棒性解耦

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。